From caplanr at predsci.com Wed Aug 1 12:07:52 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Wed, 1 Aug 2012 10:07:52 -0700 Subject: [petsc-users] Save matrix view to postscript? In-Reply-To: References: Message-ID: After configuring with --with-afterimage I now cannot even get eh x-windows to show the matrix at all. I get the following error: X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 1 (X_CreateWindow) Value in failed request: 0xae Serial number of failed request: 7 Current serial number in output stream: 14 - Ron On Tue, Jul 31, 2012 at 4:20 PM, Jed Brown wrote: > On Tue, Jul 31, 2012 at 4:13 PM, Ronald M. Caplan wrote: > >> Thanks! >> >> I can't seem to get it to work though. >> When I configure --with-afterimage it says that I need to download it. >> When I use --download-afterimage it says: >> "External package afterimage does not support --download-afterimage" >> >> I am using a MAC with OS X. Where can I download afterimage? >> > > http://www.afterstep.org/afterimage/ > > >> >> Also, in general, if I want to add a configure to petsc with ./configure, >> do I have to run ./configure with ALL my options, or just with the new ones >> I want to add/change? Will it save my previous configure? >> > > Run $PETSC_ARCH/conf/reconfigure-*.py --new-options > > >> >> Thanks again for your rapid responses! >> >> - Ron C >> >> >> On Tue, Jul 31, 2012 at 3:23 PM, Barry Smith wrote: >> >>> >>> Sorry the postscript never really worked and then the world went PDF. >>> >>> You can do what you want in PETSc 3.3 with >>> >>> ierr = PetscOptionsBool("-draw_save_movie","Make a movie from the >>> images saved","PetscDrawSetSave",movie,&movie,PETSC_NULL);CHKERRQ(ierr); >>> ierr = PetscOptionsString("-draw_save","Save graphics to >>> file","PetscDrawSetSave",filename,filename,PETSC_MAX_PATH_LEN,&save);CHKERRQ(ierr); >>> >>> /*@C >>> PetscDrawSave - Saves images produced in a PetscDraw into a file as a >>> Gif file using AfterImage >>> >>> Collective on PetscDraw >>> >>> Input Parameter: >>> + draw - the graphics context >>> . filename - name of the file, if PETSC_NULL uses name of draw object >>> - movie - produce a movie of all the images >>> >>> Options Database Command: >>> + -draw_save >>> - -draw_save_movie >>> >>> Level: intermediate >>> >>> Concepts: X windows^graphics >>> Concepts: drawing^postscript >>> Concepts: postscript^graphics >>> Concepts: drawing^Microsoft Windows >>> >>> Notes: Requires that PETSc be configured with the option >>> --with-afterimage >>> >>> >>> .seealso: PetscDrawSetFromOptions(), PetscDrawCreate(), >>> PetscDrawDestroy(), PetscDrawSave() >>> >>> Please report any problems to petsc-maint at mcs.anl.gov >>> >>> Note that you must make sure the windows being saved are not covered >>> by other windows. >>> >>> Barry >>> >>> >>> On Jul 31, 2012, at 3:17 PM, "Ronald M. Caplan" >>> wrote: >>> >>> > Hi, >>> > >>> > I saw online that back in 2006 there was a way to tell PETSc to save >>> the image in -mat_view_draw into a postscript file. Is there anyway to >>> still do this? Or to same the image into any format? Or to save a movie >>> of the residual images in ksp? >>> > >>> > Thanks, >>> > >>> > Ron C. >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 1 12:53:44 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Aug 2012 12:53:44 -0500 Subject: [petsc-users] Save matrix view to postscript? In-Reply-To: References: Message-ID: Switching discussion to petsc-maint at mcs.anl.gov so we can track down the problem. Does this error come up even without -draw_save and -draw_save_movie options? If yes then please create a new PETSC_ARCH with the exact same ./configure options but NO --with-afterimage. Does this PETSC_ARCH correctly draw X windows without the error? Barry On Aug 1, 2012, at 12:07 PM, Ronald M. Caplan wrote: > After configuring with --with-afterimage I now cannot even get eh x-windows to show the matrix at all. I get the following error: > > X Error of failed request: BadValue (integer parameter out of range for operation) > Major opcode of failed request: 1 (X_CreateWindow) > Value in failed request: 0xae > Serial number of failed request: 7 > Current serial number in output stream: 14 > > - Ron > > > > On Tue, Jul 31, 2012 at 4:20 PM, Jed Brown wrote: > On Tue, Jul 31, 2012 at 4:13 PM, Ronald M. Caplan wrote: > Thanks! > > I can't seem to get it to work though. > When I configure --with-afterimage it says that I need to download it. When I use --download-afterimage it says: > "External package afterimage does not support --download-afterimage" > > I am using a MAC with OS X. Where can I download afterimage? > > http://www.afterstep.org/afterimage/ > > > Also, in general, if I want to add a configure to petsc with ./configure, do I have to run ./configure with ALL my options, or just with the new ones I want to add/change? Will it save my previous configure? > > Run $PETSC_ARCH/conf/reconfigure-*.py --new-options > > > Thanks again for your rapid responses! > > - Ron C > > > On Tue, Jul 31, 2012 at 3:23 PM, Barry Smith wrote: > > Sorry the postscript never really worked and then the world went PDF. > > You can do what you want in PETSc 3.3 with > > ierr = PetscOptionsBool("-draw_save_movie","Make a movie from the images saved","PetscDrawSetSave",movie,&movie,PETSC_NULL);CHKERRQ(ierr); > ierr = PetscOptionsString("-draw_save","Save graphics to file","PetscDrawSetSave",filename,filename,PETSC_MAX_PATH_LEN,&save);CHKERRQ(ierr); > > /*@C > PetscDrawSave - Saves images produced in a PetscDraw into a file as a Gif file using AfterImage > > Collective on PetscDraw > > Input Parameter: > + draw - the graphics context > . filename - name of the file, if PETSC_NULL uses name of draw object > - movie - produce a movie of all the images > > Options Database Command: > + -draw_save > - -draw_save_movie > > Level: intermediate > > Concepts: X windows^graphics > Concepts: drawing^postscript > Concepts: postscript^graphics > Concepts: drawing^Microsoft Windows > > Notes: Requires that PETSc be configured with the option --with-afterimage > > > .seealso: PetscDrawSetFromOptions(), PetscDrawCreate(), PetscDrawDestroy(), PetscDrawSave() > > Please report any problems to petsc-maint at mcs.anl.gov > > Note that you must make sure the windows being saved are not covered by other windows. > > Barry > > > On Jul 31, 2012, at 3:17 PM, "Ronald M. Caplan" wrote: > > > Hi, > > > > I saw online that back in 2006 there was a way to tell PETSc to save the image in -mat_view_draw into a postscript file. Is there anyway to still do this? Or to same the image into any format? Or to save a movie of the residual images in ksp? > > > > Thanks, > > > > Ron C. > > > > From caplanr at predsci.com Wed Aug 1 12:56:19 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Wed, 1 Aug 2012 10:56:19 -0700 Subject: [petsc-users] Save matrix view to postscript? In-Reply-To: References: Message-ID: Hi, Yes, the error occurs when I use -mat_view_draw even without -draw_save. My previous pets configuration was identical to mine but without the --with-afterimage and the matrices showed up in xwindows. I downloaded and installed the latest afterimage and ffmpeg. - Ron C On Wed, Aug 1, 2012 at 10:53 AM, Barry Smith wrote: > > Switching discussion to petsc-maint at mcs.anl.gov so we can track down > the problem. > > Does this error come up even without -draw_save and -draw_save_movie > options? > > If yes then please create a new PETSC_ARCH with the exact same > ./configure options but NO --with-afterimage. Does this PETSC_ARCH > correctly draw X windows without the error? > > Barry > > > > On Aug 1, 2012, at 12:07 PM, Ronald M. Caplan wrote: > > > After configuring with --with-afterimage I now cannot even get eh > x-windows to show the matrix at all. I get the following error: > > > > X Error of failed request: BadValue (integer parameter out of range for > operation) > > Major opcode of failed request: 1 (X_CreateWindow) > > Value in failed request: 0xae > > Serial number of failed request: 7 > > Current serial number in output stream: 14 > > > > - Ron > > > > > > > > On Tue, Jul 31, 2012 at 4:20 PM, Jed Brown wrote: > > On Tue, Jul 31, 2012 at 4:13 PM, Ronald M. Caplan > wrote: > > Thanks! > > > > I can't seem to get it to work though. > > When I configure --with-afterimage it says that I need to download it. > When I use --download-afterimage it says: > > "External package afterimage does not support --download-afterimage" > > > > I am using a MAC with OS X. Where can I download afterimage? > > > > http://www.afterstep.org/afterimage/ > > > > > > Also, in general, if I want to add a configure to petsc with > ./configure, do I have to run ./configure with ALL my options, or just with > the new ones I want to add/change? Will it save my previous configure? > > > > Run $PETSC_ARCH/conf/reconfigure-*.py --new-options > > > > > > Thanks again for your rapid responses! > > > > - Ron C > > > > > > On Tue, Jul 31, 2012 at 3:23 PM, Barry Smith wrote: > > > > Sorry the postscript never really worked and then the world went PDF. > > > > You can do what you want in PETSc 3.3 with > > > > ierr = PetscOptionsBool("-draw_save_movie","Make a movie from the > images saved","PetscDrawSetSave",movie,&movie,PETSC_NULL);CHKERRQ(ierr); > > ierr = PetscOptionsString("-draw_save","Save graphics to > file","PetscDrawSetSave",filename,filename,PETSC_MAX_PATH_LEN,&save);CHKERRQ(ierr); > > > > /*@C > > PetscDrawSave - Saves images produced in a PetscDraw into a file as a > Gif file using AfterImage > > > > Collective on PetscDraw > > > > Input Parameter: > > + draw - the graphics context > > . filename - name of the file, if PETSC_NULL uses name of draw object > > - movie - produce a movie of all the images > > > > Options Database Command: > > + -draw_save > > - -draw_save_movie > > > > Level: intermediate > > > > Concepts: X windows^graphics > > Concepts: drawing^postscript > > Concepts: postscript^graphics > > Concepts: drawing^Microsoft Windows > > > > Notes: Requires that PETSc be configured with the option > --with-afterimage > > > > > > .seealso: PetscDrawSetFromOptions(), PetscDrawCreate(), > PetscDrawDestroy(), PetscDrawSave() > > > > Please report any problems to petsc-maint at mcs.anl.gov > > > > Note that you must make sure the windows being saved are not covered > by other windows. > > > > Barry > > > > > > On Jul 31, 2012, at 3:17 PM, "Ronald M. Caplan" > wrote: > > > > > Hi, > > > > > > I saw online that back in 2006 there was a way to tell PETSc to save > the image in -mat_view_draw into a postscript file. Is there anyway to > still do this? Or to same the image into any format? Or to save a movie > of the residual images in ksp? > > > > > > Thanks, > > > > > > Ron C. > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From caplanr at predsci.com Wed Aug 1 13:23:57 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Wed, 1 Aug 2012 11:23:57 -0700 Subject: [petsc-users] segfault in MatAssemblyEnd() when using large matrices on multi-core MAC OS-X In-Reply-To: References: Message-ID: Hi, Using FLUSH_ASSEMBLY periodically solved the segfault problem. I now use the following code to set the matrix: DO i=1,N IF (rank .eq. 0) THEN DO j=CSR_AJ(i)+1,CSR_AJ(i+1) call MatSetValue(A,i-1,CSR_AI(j),CSR_A(j),INSERT_VALUES,ierr) END DO END IF !Need to send out matrix periodically otherwise get a segfault !(local buffer gets full? prob on mac only?) IF (mod(i,100) .eq. 0) THEN call MatAssemblyBegin(A,MAT_FLUSH_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FLUSH_ASSEMBLY,ierr) END IF END DO !Assemble final matrix A across all cores: call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) and everything works fine. I find the error strange since I am on a single quad-core MAC, the "buffer" should never get "full"... Is this a bug? - Ron C. On Mon, Jul 30, 2012 at 3:09 PM, Jed Brown wrote: > On Mon, Jul 30, 2012 at 3:04 PM, Ronald M. Caplan wrote: > >> I seem to have solved the problem. >> >> I was storing my entire matrix on node 0 and then calling MatAssembly >> (begin and end) on all nodes (which should have worked...). >> >> Apparently I was using too much space for the buffering or the like, >> because when I change the code so each node sets its own matrix values, >> than the MatAssemblyEnd does not seg fault. >> > > Can you send the test case. It shouldn't seg-fault unless the machine runs > out of memory (and most desktop systems have overcommit, so the system will > kill arbitrary processes, not necessarily the job that did the latest > malloc. > > In practice, you should call MatAssemblyBegin(...,MAT_FLUSH_ASSEMBLY) > periodically. > > >> >> Why should this be the case? How many elements of a vector or matrix >> can a single node "set" before Assembly to distribute over all nodes? >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 1 13:36:20 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 1 Aug 2012 11:36:20 -0700 Subject: [petsc-users] segfault in MatAssemblyEnd() when using large matrices on multi-core MAC OS-X In-Reply-To: References: Message-ID: We have to reproduce or at least get a debugger trace to speculate about what went wrong in your case. Have you tried running in valgrind? On Wed, Aug 1, 2012 at 11:23 AM, Ronald M. Caplan wrote: > Hi, > > Using FLUSH_ASSEMBLY periodically solved the segfault problem. > I now use the following code to set the matrix: > > DO i=1,N > > IF (rank .eq. 0) THEN > DO j=CSR_AJ(i)+1,CSR_AJ(i+1) > call > MatSetValue(A,i-1,CSR_AI(j),CSR_A(j),INSERT_VALUES,ierr) > END DO > END IF > !Need to send out matrix periodically otherwise get a segfault > !(local buffer gets full? prob on mac only?) > IF (mod(i,100) .eq. 0) THEN > call MatAssemblyBegin(A,MAT_FLUSH_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FLUSH_ASSEMBLY,ierr) > END IF > END DO > !Assemble final matrix A across all cores: > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > > and everything works fine. > > I find the error strange since I am on a single quad-core MAC, the > "buffer" should never get "full"... Is this a bug? > > - Ron C. > > > On Mon, Jul 30, 2012 at 3:09 PM, Jed Brown wrote: > >> On Mon, Jul 30, 2012 at 3:04 PM, Ronald M. Caplan wrote: >> >>> I seem to have solved the problem. >>> >>> I was storing my entire matrix on node 0 and then calling MatAssembly >>> (begin and end) on all nodes (which should have worked...). >>> >>> Apparently I was using too much space for the buffering or the like, >>> because when I change the code so each node sets its own matrix values, >>> than the MatAssemblyEnd does not seg fault. >>> >> >> Can you send the test case. It shouldn't seg-fault unless the machine >> runs out of memory (and most desktop systems have overcommit, so the system >> will kill arbitrary processes, not necessarily the job that did the latest >> malloc. >> >> In practice, you should call MatAssemblyBegin(...,MAT_FLUSH_ASSEMBLY) >> periodically. >> >> >>> >>> Why should this be the case? How many elements of a vector or matrix >>> can a single node "set" before Assembly to distribute over all nodes? >>> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From caplanr at predsci.com Wed Aug 1 13:44:11 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Wed, 1 Aug 2012 11:44:11 -0700 Subject: [petsc-users] segfault in MatAssemblyEnd() when using large matrices on multi-core MAC OS-X In-Reply-To: References: Message-ID: Yes, but valgrind did not catch any errors. I am having the same segfault issue when I use vectors which are "too big" and do the following: !Initialize xtrue vector to random values (true solution of Ax=b): IF (rank .eq. 0) THEN allocate(xvec(N)) allocate(veci(N)) !Create list of indices (0-based for petsc) DO i=0,N-1 veci(i+1) = i END DO call RANDOM_NUMBER(xvec) call VecSetValues(xtrue,N,veci,xvec,INSERT_VALUES,ierr) deallocate(veci) END IF !Assemble the xtrue vector across all cores: call VecAssemblyBegin(xtrue,ierr); call VecAssemblyEnd(xtrue,ierr); This works on 1 processor but on multiple cores it segfaults if N is "too big" (around 3 million). Is there an equivalent "flush" command for vectors to get around this? - Ron On Wed, Aug 1, 2012 at 11:36 AM, Jed Brown wrote: > We have to reproduce or at least get a debugger trace to speculate about > what went wrong in your case. Have you tried running in valgrind? > > > On Wed, Aug 1, 2012 at 11:23 AM, Ronald M. Caplan wrote: > >> Hi, >> >> Using FLUSH_ASSEMBLY periodically solved the segfault problem. >> I now use the following code to set the matrix: >> >> DO i=1,N >> >> IF (rank .eq. 0) THEN >> DO j=CSR_AJ(i)+1,CSR_AJ(i+1) >> call >> MatSetValue(A,i-1,CSR_AI(j),CSR_A(j),INSERT_VALUES,ierr) >> END DO >> END IF >> !Need to send out matrix periodically otherwise get a segfault >> !(local buffer gets full? prob on mac only?) >> IF (mod(i,100) .eq. 0) THEN >> call MatAssemblyBegin(A,MAT_FLUSH_ASSEMBLY,ierr) >> call MatAssemblyEnd(A,MAT_FLUSH_ASSEMBLY,ierr) >> END IF >> END DO >> !Assemble final matrix A across all cores: >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> and everything works fine. >> >> I find the error strange since I am on a single quad-core MAC, the >> "buffer" should never get "full"... Is this a bug? >> >> - Ron C. >> >> >> On Mon, Jul 30, 2012 at 3:09 PM, Jed Brown wrote: >> >>> On Mon, Jul 30, 2012 at 3:04 PM, Ronald M. Caplan wrote: >>> >>>> I seem to have solved the problem. >>>> >>>> I was storing my entire matrix on node 0 and then calling MatAssembly >>>> (begin and end) on all nodes (which should have worked...). >>>> >>>> Apparently I was using too much space for the buffering or the like, >>>> because when I change the code so each node sets its own matrix values, >>>> than the MatAssemblyEnd does not seg fault. >>>> >>> >>> Can you send the test case. It shouldn't seg-fault unless the machine >>> runs out of memory (and most desktop systems have overcommit, so the system >>> will kill arbitrary processes, not necessarily the job that did the latest >>> malloc. >>> >>> In practice, you should call MatAssemblyBegin(...,MAT_FLUSH_ASSEMBLY) >>> periodically. >>> >>> >>>> >>>> Why should this be the case? How many elements of a vector or matrix >>>> can a single node "set" before Assembly to distribute over all nodes? >>>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 1 14:14:16 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 1 Aug 2012 12:14:16 -0700 Subject: [petsc-users] segfault in MatAssemblyEnd() when using large matrices on multi-core MAC OS-X In-Reply-To: References: Message-ID: On Wed, Aug 1, 2012 at 11:44 AM, Ronald M. Caplan wrote: > This works on 1 processor but on multiple cores it segfaults if N is "too > big" (around 3 million). Is there an equivalent "flush" command for > vectors to get around this? You can just call the usual VecAssemblyBegin/End. But 3 million should not be anywhere near a real problem. Can you reproduce the problem on a different computer? This may be a system/environment problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From burckhardt at itis.ethz.ch Thu Aug 2 06:58:07 2012 From: burckhardt at itis.ethz.ch (tribur) Date: Thu, 02 Aug 2012 13:58:07 +0200 Subject: [petsc-users] PCFIELDSPLIT and RHS Message-ID: <9137ca175409b7c1566b964dd0ff0679@speag.com> Dear PETsc team, We are confrontated with the problem of a complex-valued equation system M * z = (A + i*B) * (x + i*y) = b + i*c which shows slow convergence using GMRES and, e.g., Block Jacobi or SOR as preconditioner. To be able to use algebraic multigrid preconditioning, we are trying to solve the equivalent real-valued system (A -B; B A) * (x; y) = (b; c) using PCFIELDSPLIT and, e.g., -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -fieldsplit_real_ksp_type gmres -fieldsplit_real_ksp_max_it 10 -fieldsplit_real_pc_type bjacobi -fieldsplit_real_sub_pc_type lu -fieldsplit_imag_ksp_type fgmres -fieldsplit_imag_ksp_max_it 10 -fieldsplit_imag_pc_type none However, PCFIELDSPLIT does'nt work due to the lacking convergence of ksp(A) with the right hand side bTmp given by the current FGMRES iteration (outer loop). kspSolve(kspA, b, x) converges acceptably fast using, e.g., GMRES and Block Jacobi, but kspSolve(kspA, bTmp, xTmp) shows stagnation after about 3 iterations. What can we do? Do think a good initial estimate for (x,y) helps? Thanks a lot for any hints, Kathrin From knepley at gmail.com Fri Aug 3 00:10:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 00:10:49 -0500 Subject: [petsc-users] PCFIELDSPLIT and RHS In-Reply-To: <9137ca175409b7c1566b964dd0ff0679@speag.com> References: <9137ca175409b7c1566b964dd0ff0679@speag.com> Message-ID: On Thu, Aug 2, 2012 at 6:58 AM, tribur wrote: > Dear PETsc team, > > We are confrontated with the problem of a complex-valued equation system M > * z = (A + i*B) * (x + i*y) = b + i*c which shows slow convergence using > GMRES and, e.g., Block Jacobi or SOR as preconditioner. > > To be able to use algebraic multigrid preconditioning, we are trying to > solve the equivalent real-valued system (A -B; B A) * (x; y) = (b; c) using > PCFIELDSPLIT and, e.g., > -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type full -fieldsplit_real_ksp_type gmres > -fieldsplit_real_ksp_max_it 10 -fieldsplit_real_pc_type bjacobi > -fieldsplit_real_sub_pc_type lu -fieldsplit_imag_ksp_type fgmres > -fieldsplit_imag_ksp_max_it 10 -fieldsplit_imag_pc_type none > > However, PCFIELDSPLIT does'nt work due to the lacking convergence of > ksp(A) with the right hand side bTmp given by the current FGMRES iteration > (outer loop). kspSolve(kspA, b, x) converges acceptably fast using, e.g., > GMRES and Block Jacobi, but kspSolve(kspA, bTmp, xTmp) shows stagnation > after about 3 iterations. > Always check with LU for all solves first, and then back off. This behavior sounds like you have a singular A. Matt > What can we do? Do think a good initial estimate for (x,y) helps? > > Thanks a lot for any hints, > > Kathrin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Aug 3 02:03:28 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 03 Aug 2012 09:03:28 +0200 Subject: [petsc-users] Recommended unknowns per processor Message-ID: <501B77C0.4070905@gmail.com> Hi, I remembered that in one of the mails, someone mentions the recommended unknowns on each processor when doing parallel solving. However, I can't find that mail. Can someone enlighten me? Also, is this number problem dependent? Btw, I'm using PETSc to solve linear equations using BCGS. Thanks! -- Yours sincerely, TAY wee-beng From thomas.huxhorn at web.de Fri Aug 3 04:10:37 2012 From: thomas.huxhorn at web.de (thomas) Date: Fri, 03 Aug 2012 11:10:37 +0200 Subject: [petsc-users] Recommended unknowns per processor In-Reply-To: <501B77C0.4070905@gmail.com> References: <501B77C0.4070905@gmail.com> Message-ID: <501B958D.3070304@web.de> Hi, look at the FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel "There must be enough work for each process to overweigh the communication time. We recommend an absolute minimum of about 10,000 unknowns per process, better is 20,000 or more." Regards Thomas Huxhorn On 08/03/2012 09:03 AM, TAY wee-beng wrote: > Hi, > > I remembered that in one of the mails, someone mentions the recommended > unknowns on each processor when doing parallel solving. > > However, I can't find that mail. Can someone enlighten me? Also, is this > number problem dependent? > > Btw, I'm using PETSc to solve linear equations using BCGS. > > Thanks! > From zonexo at gmail.com Fri Aug 3 04:52:58 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 03 Aug 2012 11:52:58 +0200 Subject: [petsc-users] Recommended unknowns per processor In-Reply-To: <501B958D.3070304@web.de> References: <501B77C0.4070905@gmail.com> <501B958D.3070304@web.de> Message-ID: <501B9F7A.2040607@gmail.com> On 3/8/2012 11:10 AM, thomas wrote: > Hi, > look at the FAQ > > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel > > "There must be enough work for each process to overweigh the > communication time. We recommend an absolute minimum of about 10,000 > unknowns per process, better is 20,000 or more." Hi Thomas, Thanks for the info. Btw, is there an upper limit? When the unknowns are 100,000 or more per process, is it better (as in faster) to use more processes instead (e.g. 20,000 per process)? > > Regards > Thomas Huxhorn > > On 08/03/2012 09:03 AM, TAY wee-beng wrote: >> Hi, >> >> I remembered that in one of the mails, someone mentions the recommended >> unknowns on each processor when doing parallel solving. >> >> However, I can't find that mail. Can someone enlighten me? Also, is this >> number problem dependent? >> >> Btw, I'm using PETSc to solve linear equations using BCGS. >> >> Thanks! >> > From knepley at gmail.com Fri Aug 3 07:05:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 07:05:47 -0500 Subject: [petsc-users] Recommended unknowns per processor In-Reply-To: <501B9F7A.2040607@gmail.com> References: <501B77C0.4070905@gmail.com> <501B958D.3070304@web.de> <501B9F7A.2040607@gmail.com> Message-ID: On Fri, Aug 3, 2012 at 4:52 AM, TAY wee-beng wrote: > On 3/8/2012 11:10 AM, thomas wrote: > >> Hi, >> look at the FAQ >> >> http://www.mcs.anl.gov/petsc/**documentation/faq.html#**slowerparallel >> >> "There must be enough work for each process to overweigh the >> communication time. We recommend an absolute minimum of about 10,000 >> unknowns per process, better is 20,000 or more." >> > > Hi Thomas, > > Thanks for the info. > > Btw, is there an upper limit? When the unknowns are 100,000 or more per > process, is it better (as in faster) to use more processes instead (e.g. > 20,000 per process)? That depends on the machine. The lower limit is the minimum number of unknowns to cover the latency of network communication. Matt > >> Regards >> Thomas Huxhorn >> >> On 08/03/2012 09:03 AM, TAY wee-beng wrote: >> >>> Hi, >>> >>> I remembered that in one of the mails, someone mentions the recommended >>> unknowns on each processor when doing parallel solving. >>> >>> However, I can't find that mail. Can someone enlighten me? Also, is this >>> number problem dependent? >>> >>> Btw, I'm using PETSc to solve linear equations using BCGS. >>> >>> Thanks! >>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Fri Aug 3 07:28:23 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Fri, 3 Aug 2012 07:28:23 -0500 Subject: [petsc-users] with hypre Message-ID: Hi All, I am trying to solve system of PDEs on AMR grid. I using PETSc 's Krylov solver for that. This requires me to use a FAC preconditioner with a non-Krylov solver for the level 0. I actually tried PETSc's Krylov Solver for level 0 (by building MPIAIJ Mat and Vec), but it does not really improve the convergence rate. So I believe I will have to use a multigrid solver for level 0. Can I use my already built PETSc Mat and Vec and call multigrid solvers from PETSc? If thats not possible, I am planning to use hypre's "SysPFMG" to write a multigrid solver for level 0. I have installed PETSc with hypre by using --download-hypre=1 option. I can build and run many examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but not ex9; which is the closest I could find for my purpose. http://acts.nersc.gov/hypre/example09/ex9.c.html I am getting some linkage error like $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In function `hypre_BoomerAMGFitVectors': $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: undefined reference to `dgels_' $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In function `hypre_BoomerAMGRelax': $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: undefined reference to `dgetrf_' $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: undefined reference to `dgetrs_' $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In function `hypre_AMGNodalSchwarzSmoother': $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: undefined reference to `dpotrf_' Any help will be appreciated. Thanks, -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 3 07:47:18 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 07:47:18 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: > Hi All, > > I am trying to solve system of PDEs on AMR grid. I using PETSc 's Krylov > solver for that. This requires me > to use a FAC preconditioner with a non-Krylov solver for the level 0. I > actually tried PETSc's Krylov Solver > for level 0 (by building MPIAIJ Mat and Vec), but it does not really > improve the convergence rate. So I believe > I will have to use a multigrid solver for level 0. Can I use my already > built PETSc Mat and Vec and call multigrid solvers from PETSc? > Use AMG to start here since it is easy. You can either use ML or GAMG (which is builtin) to start, or Hypre if you straighten out the link (likely broken from not using the makefiles). > If thats not possible, I am planning to use hypre's "SysPFMG" to write a > multigrid solver for level 0. > I have installed PETSc with hypre by using --download-hypre=1 option. I > can build and run many > examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but not > ex9; which is the > closest I could find for my purpose. > http://acts.nersc.gov/hypre/example09/ex9.c.html > > I am getting some linkage error like > Always send the entire output, including the command that was run. I have no idea what file you are trying to compile here. Does the example work? Matt > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In > function `hypre_BoomerAMGFitVectors': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: > undefined reference to `dgels_' > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In > function `hypre_BoomerAMGRelax': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: > undefined reference to `dgetrf_' > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: > undefined reference to `dgetrs_' > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In > function `hypre_AMGNodalSchwarzSmoother': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: > undefined reference to `dpotrf_' > > Any help will be appreciated. > > Thanks, > -- > Amneet > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Fri Aug 3 08:12:28 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Fri, 3 Aug 2012 08:12:28 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: Thanks Matt for a quick reply. Attached is the Makefile I used for building ex9. I used the same Makefile provided by hypre. I just changed the path of hypre's directory. Also attached is the command and full output I got. Using the same Makefile I was able to build and run ex1, ex2, ex3 and ex4 of hypre. Can you point me to PETSc's examples of AMG, ML or GAMG? On Fri, Aug 3, 2012 at 7:47 AM, Matthew Knepley wrote: > On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: > >> Hi All, >> >> I am trying to solve system of PDEs on AMR grid. I using PETSc 's Krylov >> solver for that. This requires me >> to use a FAC preconditioner with a non-Krylov solver for the level 0. I >> actually tried PETSc's Krylov Solver >> for level 0 (by building MPIAIJ Mat and Vec), but it does not really >> improve the convergence rate. So I believe >> I will have to use a multigrid solver for level 0. Can I use my already >> built PETSc Mat and Vec and call multigrid solvers from PETSc? >> > > Use AMG to start here since it is easy. You can either use ML or GAMG > (which is builtin) to start, or Hypre if > you straighten out the link (likely broken from not using the makefiles). > > >> If thats not possible, I am planning to use hypre's "SysPFMG" to write a >> multigrid solver for level 0. >> I have installed PETSc with hypre by using --download-hypre=1 option. I >> can build and run many >> examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but >> not ex9; which is the >> closest I could find for my purpose. >> http://acts.nersc.gov/hypre/example09/ex9.c.html >> >> I am getting some linkage error like >> > > Always send the entire output, including the command that was run. I have > no idea what file you are trying to > compile here. Does the example work? > > Matt > > >> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In >> function `hypre_BoomerAMGFitVectors': >> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: >> undefined reference to `dgels_' >> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In >> function `hypre_BoomerAMGRelax': >> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: >> undefined reference to `dgetrf_' >> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: >> undefined reference to `dgetrs_' >> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In >> function `hypre_AMGNodalSchwarzSmoother': >> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: >> undefined reference to `dpotrf_' >> >> Any help will be appreciated. >> >> Thanks, >> -- >> Amneet >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: commandToRun Type: application/octet-stream Size: 6508 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 8052 bytes Desc: not available URL: From knepley at gmail.com Fri Aug 3 08:24:19 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 08:24:19 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Fri, Aug 3, 2012 at 8:12 AM, Amneet Bhalla wrote: > Thanks Matt for a quick reply. > > Attached is the Makefile I used for building ex9. I used the same Makefile > provided by hypre. I just changed the > path of hypre's directory. Also attached is the command and full output I > got. Using the same Makefile I was able to > build and run ex1, ex2, ex3 and ex4 of hypre. > You are missing the BLAS. Use the PETSc makefiles. > Can you point me to PETSc's examples of AMG, ML or GAMG? > cd src/snes/examples/tutorials make ex5 ./ex5 -pc_type ml -snes_monitor Matt > On Fri, Aug 3, 2012 at 7:47 AM, Matthew Knepley wrote: > >> On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: >> >>> Hi All, >>> >>> I am trying to solve system of PDEs on AMR grid. I using PETSc 's Krylov >>> solver for that. This requires me >>> to use a FAC preconditioner with a non-Krylov solver for the level 0. I >>> actually tried PETSc's Krylov Solver >>> for level 0 (by building MPIAIJ Mat and Vec), but it does not really >>> improve the convergence rate. So I believe >>> I will have to use a multigrid solver for level 0. Can I use my already >>> built PETSc Mat and Vec and call multigrid solvers from PETSc? >>> >> >> Use AMG to start here since it is easy. You can either use ML or GAMG >> (which is builtin) to start, or Hypre if >> you straighten out the link (likely broken from not using the makefiles). >> >> >>> If thats not possible, I am planning to use hypre's "SysPFMG" to write a >>> multigrid solver for level 0. >>> I have installed PETSc with hypre by using --download-hypre=1 option. I >>> can build and run many >>> examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but >>> not ex9; which is the >>> closest I could find for my purpose. >>> http://acts.nersc.gov/hypre/example09/ex9.c.html >>> >>> I am getting some linkage error like >>> >> >> Always send the entire output, including the command that was run. I have >> no idea what file you are trying to >> compile here. Does the example work? >> >> Matt >> >> >>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In >>> function `hypre_BoomerAMGFitVectors': >>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: >>> undefined reference to `dgels_' >>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In >>> function `hypre_BoomerAMGRelax': >>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: >>> undefined reference to `dgetrf_' >>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: >>> undefined reference to `dgetrs_' >>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In >>> function `hypre_AMGNodalSchwarzSmoother': >>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: >>> undefined reference to `dpotrf_' >>> >>> Any help will be appreciated. >>> >>> Thanks, >>> -- >>> Amneet >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Amneet > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Aug 3 09:05:40 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 3 Aug 2012 10:05:40 -0400 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: <0554D2E9-F887-4DEF-8838-445630C1A25B@columbia.edu> For GAMG you can use ex54, ex55, ex54f, ex56: cd src/ksp/ksp/examples/tutorials make ex56 make runex56 Mark On Aug 3, 2012, at 9:12 AM, Amneet Bhalla wrote: > Thanks Matt for a quick reply. > > Attached is the Makefile I used for building ex9. I used the same Makefile provided by hypre. I just changed the > path of hypre's directory. Also attached is the command and full output I got. Using the same Makefile I was able to > build and run ex1, ex2, ex3 and ex4 of hypre. > > Can you point me to PETSc's examples of AMG, ML or GAMG? > > On Fri, Aug 3, 2012 at 7:47 AM, Matthew Knepley wrote: > On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: > Hi All, > > I am trying to solve system of PDEs on AMR grid. I using PETSc 's Krylov solver for that. This requires me > to use a FAC preconditioner with a non-Krylov solver for the level 0. I actually tried PETSc's Krylov Solver > for level 0 (by building MPIAIJ Mat and Vec), but it does not really improve the convergence rate. So I believe > I will have to use a multigrid solver for level 0. Can I use my already built PETSc Mat and Vec and call multigrid solvers from PETSc? > > Use AMG to start here since it is easy. You can either use ML or GAMG (which is builtin) to start, or Hypre if > you straighten out the link (likely broken from not using the makefiles). > > If thats not possible, I am planning to use hypre's "SysPFMG" to write a multigrid solver for level 0. > I have installed PETSc with hypre by using --download-hypre=1 option. I can build and run many > examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but not ex9; which is the > closest I could find for my purpose. > http://acts.nersc.gov/hypre/example09/ex9.c.html > > I am getting some linkage error like > > Always send the entire output, including the command that was run. I have no idea what file you are trying to > compile here. Does the example work? > > Matt > > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In function `hypre_BoomerAMGFitVectors': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: undefined reference to `dgels_' > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In function `hypre_BoomerAMGRelax': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: undefined reference to `dgetrf_' > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: undefined reference to `dgetrs_' > $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In function `hypre_AMGNodalSchwarzSmoother': > $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: undefined reference to `dpotrf_' > > Any help will be appreciated. > > Thanks, > -- > Amneet > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > -- > Amneet > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Fri Aug 3 09:18:13 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Fri, 3 Aug 2012 22:18:13 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? Message-ID: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? Thank you! Sincerely, Zeng Xiangze -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 3 09:28:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 09:28:07 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> Message-ID: On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: > Dear all, > > When I use the CPU solve the equations, it takes 78 minutes, when I change > to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper > say when using PETCs with GPU to solve the large sparse matrix equations, > it can be several times faster? What's the matter? > For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt > Thank you! > > Sincerely, > Zeng Xiangze > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Aug 3 11:48:01 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 03 Aug 2012 18:48:01 +0200 Subject: [petsc-users] Recommended unknowns per processor In-Reply-To: References: <501B77C0.4070905@gmail.com> <501B958D.3070304@web.de> <501B9F7A.2040607@gmail.com> Message-ID: <501C00C1.6000404@gmail.com> On 3/8/2012 2:05 PM, Matthew Knepley wrote: > On Fri, Aug 3, 2012 at 4:52 AM, TAY wee-beng > wrote: > > On 3/8/2012 11:10 AM, thomas wrote: > > Hi, > look at the FAQ > > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel > > "There must be enough work for each process to overweigh the > communication time. We recommend an absolute minimum of about > 10,000 unknowns per process, better is 20,000 or more." > > > Hi Thomas, > > Thanks for the info. > > Btw, is there an upper limit? When the unknowns are 100,000 or > more per process, is it better (as in faster) to use more > processes instead (e.g. 20,000 per process)? > > > That depends on the machine. The lower limit is the minimum number of > unknowns to cover the latency of network communication. Thanks for the info! > > Matt > > > Regards > Thomas Huxhorn > > On 08/03/2012 09:03 AM, TAY wee-beng wrote: > > Hi, > > I remembered that in one of the mails, someone mentions > the recommended > unknowns on each processor when doing parallel solving. > > However, I can't find that mail. Can someone enlighten me? > Also, is this > number problem dependent? > > Btw, I'm using PETSc to solve linear equations using BCGS. > > Thanks! > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Fri Aug 3 11:52:27 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Fri, 3 Aug 2012 11:52:27 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: I am not sure which PETSc makefile to use as there are many in PETSc directory. Also do you mean PETSc makefile just to build this example or to build hypre from scratch? In the makefile I sent you I added the path of libfblas and libflapack which PETSc had downloaded and installed as an external package but it didn't work either. On Fri, Aug 3, 2012 at 8:24 AM, Matthew Knepley wrote: > On Fri, Aug 3, 2012 at 8:12 AM, Amneet Bhalla wrote: > >> Thanks Matt for a quick reply. >> >> Attached is the Makefile I used for building ex9. I used the same >> Makefile provided by hypre. I just changed the >> path of hypre's directory. Also attached is the command and full output I >> got. Using the same Makefile I was able to >> build and run ex1, ex2, ex3 and ex4 of hypre. >> > > You are missing the BLAS. Use the PETSc makefiles. > > >> Can you point me to PETSc's examples of AMG, ML or GAMG? >> > > cd src/snes/examples/tutorials > make ex5 > ./ex5 -pc_type ml -snes_monitor > > Matt > > >> On Fri, Aug 3, 2012 at 7:47 AM, Matthew Knepley wrote: >> >>> On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: >>> >>>> Hi All, >>>> >>>> I am trying to solve system of PDEs on AMR grid. I using PETSc 's >>>> Krylov solver for that. This requires me >>>> to use a FAC preconditioner with a non-Krylov solver for the level 0. I >>>> actually tried PETSc's Krylov Solver >>>> for level 0 (by building MPIAIJ Mat and Vec), but it does not really >>>> improve the convergence rate. So I believe >>>> I will have to use a multigrid solver for level 0. Can I use my already >>>> built PETSc Mat and Vec and call multigrid solvers from PETSc? >>>> >>> >>> Use AMG to start here since it is easy. You can either use ML or GAMG >>> (which is builtin) to start, or Hypre if >>> you straighten out the link (likely broken from not using the makefiles). >>> >>> >>>> If thats not possible, I am planning to use hypre's "SysPFMG" to write >>>> a multigrid solver for level 0. >>>> I have installed PETSc with hypre by using --download-hypre=1 option. >>>> I can build and run many >>>> examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but >>>> not ex9; which is the >>>> closest I could find for my purpose. >>>> http://acts.nersc.gov/hypre/example09/ex9.c.html >>>> >>>> I am getting some linkage error like >>>> >>> >>> Always send the entire output, including the command that was run. I >>> have no idea what file you are trying to >>> compile here. Does the example work? >>> >>> Matt >>> >>> >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In >>>> function `hypre_BoomerAMGFitVectors': >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: >>>> undefined reference to `dgels_' >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In >>>> function `hypre_BoomerAMGRelax': >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: >>>> undefined reference to `dgetrf_' >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: >>>> undefined reference to `dgetrs_' >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In >>>> function `hypre_AMGNodalSchwarzSmoother': >>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: >>>> undefined reference to `dpotrf_' >>>> >>>> Any help will be appreciated. >>>> >>>> Thanks, >>>> -- >>>> Amneet >>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Amneet >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 3 12:08:13 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 12:08:13 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Fri, Aug 3, 2012 at 11:52 AM, Amneet Bhalla wrote: > I am not sure which PETSc makefile to use as there are many in PETSc > directory. Also do you mean PETSc makefile just to build this example or to > build hypre from scratch? In the makefile I sent you I added the path of > libfblas and libflapack which PETSc had downloaded and installed as an > external package but it didn't work either. > There is an entire chapter in the manual on makefiles, including a section on making user makefiles. Have you read this? You can see our full link line using "make getlinklibs", but that should be used by advanced users of make. Matt > On Fri, Aug 3, 2012 at 8:24 AM, Matthew Knepley wrote: > >> On Fri, Aug 3, 2012 at 8:12 AM, Amneet Bhalla wrote: >> >>> Thanks Matt for a quick reply. >>> >>> Attached is the Makefile I used for building ex9. I used the same >>> Makefile provided by hypre. I just changed the >>> path of hypre's directory. Also attached is the command and full output >>> I got. Using the same Makefile I was able to >>> build and run ex1, ex2, ex3 and ex4 of hypre. >>> >> >> You are missing the BLAS. Use the PETSc makefiles. >> >> >>> Can you point me to PETSc's examples of AMG, ML or GAMG? >>> >> >> cd src/snes/examples/tutorials >> make ex5 >> ./ex5 -pc_type ml -snes_monitor >> >> Matt >> >> >>> On Fri, Aug 3, 2012 at 7:47 AM, Matthew Knepley wrote: >>> >>>> On Fri, Aug 3, 2012 at 7:28 AM, Amneet Bhalla wrote: >>>> >>>>> Hi All, >>>>> >>>>> I am trying to solve system of PDEs on AMR grid. I using PETSc 's >>>>> Krylov solver for that. This requires me >>>>> to use a FAC preconditioner with a non-Krylov solver for the level 0. >>>>> I actually tried PETSc's Krylov Solver >>>>> for level 0 (by building MPIAIJ Mat and Vec), but it does not really >>>>> improve the convergence rate. So I believe >>>>> I will have to use a multigrid solver for level 0. Can I use my >>>>> already built PETSc Mat and Vec and call multigrid solvers from PETSc? >>>>> >>>> >>>> Use AMG to start here since it is easy. You can either use ML or GAMG >>>> (which is builtin) to start, or Hypre if >>>> you straighten out the link (likely broken from not using the >>>> makefiles). >>>> >>>> >>>>> If thats not possible, I am planning to use hypre's "SysPFMG" to write >>>>> a multigrid solver for level 0. >>>>> I have installed PETSc with hypre by using --download-hypre=1 option. >>>>> I can build and run many >>>>> examples in PETSC_ARCH/externalpackages/hypre-2.7.0b/src/examples/ but >>>>> not ex9; which is the >>>>> closest I could find for my purpose. >>>>> http://acts.nersc.gov/hypre/example09/ex9.c.html >>>>> >>>>> I am getting some linkage error like >>>>> >>>> >>>> Always send the entire output, including the command that was run. I >>>> have no idea what file you are trying to >>>> compile here. Does the example work? >>>> >>>> Matt >>>> >>>> >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_gsmg.o): In >>>>> function `hypre_BoomerAMGFitVectors': >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_gsmg.c:735: >>>>> undefined reference to `dgels_' >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(par_relax.o): In >>>>> function `hypre_BoomerAMGRelax': >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3135: >>>>> undefined reference to `dgetrf_' >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/par_relax.c:3146: >>>>> undefined reference to `dgetrs_' >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/linux/lib/libHYPRE.a(schwarz.o): In >>>>> function `hypre_AMGNodalSchwarzSmoother': >>>>> $HOME/SOFTWARES/LIBRARIES/PETSc/externalpackages/hypre-2.7.0b/src/parcsr_ls/schwarz.c:312: >>>>> undefined reference to `dpotrf_' >>>>> >>>>> Any help will be appreciated. >>>>> >>>>> Thanks, >>>>> -- >>>>> Amneet >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Amneet >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Amneet > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Fri Aug 3 13:34:46 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Fri, 3 Aug 2012 13:34:46 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: Thanks Matt for pointing to Makefile manual. I just included these lines include $(PETSC_DIR)/conf/variables LIBS = $(PETSC_LIB) in the hypre's makefile and it is now working. This issue is resolved. I will try to write preconditioner using hypre's multigrid solvers. Example files which you pointed for AMG and ML are for non-linear solvers and distributed arrays. The equation I am solving is linear for which I have a sparse matrix MPIAIJ. Is it possible to use multigrid solvers in PETSc without going through the process of again creating new arrays/matrices? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 3 14:03:21 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 14:03:21 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Fri, Aug 3, 2012 at 1:34 PM, Amneet Bhalla wrote: > Thanks Matt for pointing to Makefile manual. I just included these lines > > include $(PETSC_DIR)/conf/variables > LIBS = $(PETSC_LIB) > > in the hypre's makefile and it is now working. This issue is resolved. I > will try to write preconditioner using hypre's multigrid solvers. > > Example files which you pointed for AMG and ML are for non-linear solvers > and distributed arrays. The > equation I am solving is linear for which I have a sparse matrix MPIAIJ. > Is it possible to use multigrid solvers > in PETSc without going through the process of again creating new > arrays/matrices? > The point is that if you just use -pc_type gamg from the command line it is no different for these examples. They also build sparse matrices underneath. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 3 14:21:18 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Aug 2012 14:21:18 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Aug 3, 2012, at 1:34 PM, Amneet Bhalla wrote: > Thanks Matt for pointing to Makefile manual. I just included these lines > > include $(PETSC_DIR)/conf/variables > LIBS = $(PETSC_LIB) > > in the hypre's makefile and it is now working. This issue is resolved. I will try to write preconditioner using hypre's multigrid solvers. > > Example files which you pointed for AMG and ML are for non-linear solvers and distributed arrays. The > equation I am solving is linear for which I have a sparse matrix MPIAIJ. Is it possible to use multigrid solvers > in PETSc without going through the process of again creating new arrays/matrices? Yup, that is exactly what the hyper BoomerAMG, Trilinos ML and PETSc gang solvers are for. Barry > > From mail2amneet at gmail.com Fri Aug 3 14:31:51 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Fri, 3 Aug 2012 14:31:51 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: I see... Do I need to have some external package for gamg? I am getting this error Unable to find requested PC type gamg! Also its not listed in -pc_type options when I do -help On Fri, Aug 3, 2012 at 2:21 PM, Barry Smith wrote: > > On Aug 3, 2012, at 1:34 PM, Amneet Bhalla wrote: > > > Thanks Matt for pointing to Makefile manual. I just included these lines > > > > include $(PETSC_DIR)/conf/variables > > LIBS = $(PETSC_LIB) > > > > in the hypre's makefile and it is now working. This issue is resolved. I > will try to write preconditioner using hypre's multigrid solvers. > > > > Example files which you pointed for AMG and ML are for non-linear > solvers and distributed arrays. The > > equation I am solving is linear for which I have a sparse matrix MPIAIJ. > Is it possible to use multigrid solvers > > in PETSc without going through the process of again creating new > arrays/matrices? > > Yup, that is exactly what the hyper BoomerAMG, Trilinos ML and PETSc > gang solvers are for. > > Barry > > > > > > > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 3 14:36:10 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Aug 2012 14:36:10 -0500 Subject: [petsc-users] with hypre In-Reply-To: References: Message-ID: On Aug 3, 2012, at 2:31 PM, Amneet Bhalla wrote: > I see... Do I need to have some external package for gamg? I am getting this error > > Unable to find requested PC type gamg! > > Also its not listed in -pc_type options when I do -help You should be using PTESc 3.3 or petsc-dev Barry > > > On Fri, Aug 3, 2012 at 2:21 PM, Barry Smith wrote: > > On Aug 3, 2012, at 1:34 PM, Amneet Bhalla wrote: > > > Thanks Matt for pointing to Makefile manual. I just included these lines > > > > include $(PETSC_DIR)/conf/variables > > LIBS = $(PETSC_LIB) > > > > in the hypre's makefile and it is now working. This issue is resolved. I will try to write preconditioner using hypre's multigrid solvers. > > > > Example files which you pointed for AMG and ML are for non-linear solvers and distributed arrays. The > > equation I am solving is linear for which I have a sparse matrix MPIAIJ. Is it possible to use multigrid solvers > > in PETSc without going through the process of again creating new arrays/matrices? > > Yup, that is exactly what the hyper BoomerAMG, Trilinos ML and PETSc gang solvers are for. > > Barry > > > > > > > > > > -- > Amneet > > > > From hadsed at gmail.com Fri Aug 3 15:44:31 2012 From: hadsed at gmail.com (Hadayat Seddiqi) Date: Fri, 3 Aug 2012 16:44:31 -0400 Subject: [petsc-users] Integrating PETSc with existing software using CMake In-Reply-To: References: Message-ID: I found out that the permissions in my working directory were out of whack, which wasn't allowing the script to be run. I've fixed this and everything works flawlessly. Thanks Jed for the CMake scripts. On Mon, Jul 30, 2012 at 7:46 PM, Jed Brown wrote: > On Mon, Jul 30, 2012 at 4:27 PM, Hadayat Seddiqi wrote: > >> Hi Jed, >> >> Apparently it's failing to run the test script (it says >> PETSC_EXECUTABLE_RUNS was not set). If I'm correct, the output of the >> failed test script should be in CMakeFiles/CMakeError.log . Looking at >> this, it gives me the following: >> >> In file included from /home/h37/petsc-3.2-p7/include/petscis.h:7, >> from /home/h37/petsc-3.2-p7/include/petscvec.h:9, >> from /home/h37/petsc-3.2-p7/include/petscmat.h:6, >> from /home/h37/petsc-3.2-p7/include/petscdm.h:6, >> from /home/h37/petsc-3.2-p7/include/petscpc.h:6, >> from /home/h37/petsc-3.2-p7/include/petscksp.h:6, >> from /home/h37/petsc-3.2-p7/include/petscsnes.h:6, >> from /home/h37/petsc-3.2-p7/include/petscts.h:7, >> from >> /home/h37/sapphiresimulator/FOP/build/CMakeFiles/CMakeTmp/src.c:3: >> /home/h37/petsc-3.2-p7/include/petscsys.h:105:17: error: mpi.h: No such >> file or director >> > > You may need to set the correct MPI wrapper compiler. If that doesn't > work, can you send the full Output and Error files to petsc-maint? > > >> >> And of course a lot more afterwards. How is it possible that it cannot >> find MPI, even though all of the test cases and example/tutorial programs >> worked with mpi? >> >> I realize it's probably bad practice, but I also thought I would try to >> disable this flag by forcing PETSC_EXECUTABLE_RUNS to "YES", but it doesn't >> seem to work. I tried commenting out if (${${run}}) and the corresponding >> endif, as well as moving it outside the macro. I feel rather silly about >> this, but I just cannot get it to work. >> >> Thanks, >> >> Had >> >> On Mon, Jul 30, 2012 at 5:36 PM, Jed Brown wrote: >> >>> On Mon, Jul 30, 2012 at 4:19 PM, Hadayat Seddiqi wrote: >>> >>>> Hello, >>>> >>>> I'm working on a large numerical software project whose framework has >>>> largely been developed already. We're using CMake to generate makefiles. >>>> I'm also using SLEPc (for full disclosure). The examples given by PETSc and >>>> SLEPc documentation require me to include makefiles, but I don't know of >>>> any straightforward way to command CMake to do this for me. >>>> >>>> I have looked at the FAQ's link for the CMake question: >>>> https://github.com/jedbrown/dohp But this seems very old, and in any >>>> case it doesn't exactly work. >>>> >>> >>> The CMake stuff is here. >>> >>> https://github.com/jedbrown/cmake-modules/ >>> >>> I'm not an expert on CMake, so I couldn't say what was the causing the >>>> problem, but in the end it told me it could not find the PETSc libraries. >>>> It seemed to be rather complicated-- I know PETSc will be where I need it, >>>> so I don't need all the verification that it's there and everything works. >>>> I thought, with the benefit of more intimate knowledge of how PETSc runs, >>>> that someone could show a much simpler way (it seems to me that this ought >>>> to be the case). >>>> >>> >>> The problem is that there are lots of ways that things can "not work", >>> so its important for the FindPETSc.cmake script to really try. Also, CMake >>> insists on taking parameters in a different way (e.g. converting >>> command-line flags to full paths). >>> >>> Have you looked at the logs (CMakeFiles/CMake{Output,Error}.log >>> >>> Here is more active package that uses the FindPETSc.cmake script >>> >>> https://github.com/pism/pism >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Fri Aug 3 17:38:15 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Fri, 3 Aug 2012 15:38:15 -0700 Subject: [petsc-users] error when using ksplsqr for overdetermined system Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018F1A5819CF@EMAIL04.pnl.gov> Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. I have read the following threads regarding the topic: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html I defined a 4*3 matrix A and a vector b in sparse CSR format: irow: 1 3 5 6 9 icol: 1 2 1 3 2 1 2 3 ival: 1 2 2 5 3 1 4 1 rhs: 1 2 3 4 A = 1 2 0 2 0 5 0 3 0 1 4 1 b = 1 2 3 4 I wrote the following code to solve this system: static char help[]="Reading in a matrix\n"; #include #include #include #include "petscvec.h" /* This enables us to use vectors. */ #include "petscmat.h" /* This enables us to use Matrices. It includes the petscvec header file*/ #include "petscksp.h" /* Now we can solve linear systems. Solvers used are KSP. */ int main(int argc, char **argv) { /* Declaration of Matrix A and some vectors x*/ Vec b,x; Mat A; FILE *fp; MPI_Comm comm; PetscInt n=4,m=3,nz=8,index; PetscErrorCode ierr; PetscInt i,j; comm = MPI_COMM_SELF; PetscInt irow[n+1],icol[nz]; PetscScalar ival[nz],rhs[n]; PetscInt *cnt,*col; KSP ksp; /* This part is needed for the help flag supplied at run-time*/ ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); /*===================================================================================*/ fp=fopen("irow.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < n+1; i++) { if (fscanf(fp,"%d", &irow[i]) != 1) { fprintf(stderr, "Failed to read irow vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); fp=fopen("icol.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%d", &icol[i]) != 1) { fprintf(stderr, "Failed to read icol vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); fp=fopen("ival.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%lf", &ival[i]) != 1) { fprintf(stderr, "Failed to read ival vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); /*===================================================================================*/ // determine number of nonzeros per row in the new matrix PetscMalloc((n+1)*sizeof(PetscInt),&cnt); for (i=0;i From knepley at gmail.com Fri Aug 3 18:35:36 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Aug 2012 18:35:36 -0500 Subject: [petsc-users] error when using ksplsqr for overdetermined system In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018F1A5819CF@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018F1A5819CF@EMAIL04.pnl.gov> Message-ID: On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang wrote: > Hi, all, I?m trying to use ksplsqr to solve an overdetermined system. > I have read the following threads regarding the topic: > *http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html* > *http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html* > > I defined a 4*3 matrix A and a vector b in sparse CSR format: > irow: 1 3 5 6 9 > icol: 1 2 1 3 2 1 2 3 > ival: 1 2 2 5 3 1 4 1 > rhs: 1 2 3 4 > > A = 1 2 0 > 2 0 5 > 0 3 0 > 1 4 1 > b = 1 > 2 > 3 > 4 > > I wrote the following code to solve this system: > > static char help[]="Reading in a matrix\n"; > > #include > #include > #include > #include "petscvec.h" /* This enables us to use vectors. */ > #include "petscmat.h" /* This enables us to use Matrices. It includes the > petscvec header file*/ > #include "petscksp.h" /* Now we can solve linear systems. Solvers used are > KSP. */ > > int main(int argc, char **argv) > { > /* Declaration of Matrix A and some vectors x*/ > Vec b,x; > Mat A; > FILE *fp; > > MPI_Comm comm; > PetscInt n=4,m=3,nz=8,index; > PetscErrorCode ierr; > PetscInt i,j; > comm = MPI_COMM_SELF; > PetscInt irow[n+1],icol[nz]; > PetscScalar ival[nz],rhs[n]; > PetscInt *cnt,*col; > > KSP ksp; > > /* This part is needed for the help flag supplied at run-time*/ > ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); > ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); > > > /*===================================================================================*/ > > fp=fopen("irow.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n+1; i++) > { > if (fscanf(fp,"%d", &irow[i]) != 1) > { > fprintf(stderr, "Failed to read irow vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); > > fp=fopen("icol.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%d", &icol[i]) != 1) > { > fprintf(stderr, "Failed to read icol vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); > > fp=fopen("ival.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%lf", &ival[i]) != 1) > { > fprintf(stderr, "Failed to read ival vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); > > > /*===================================================================================*/ > > // determine number of nonzeros per row in the new matrix > PetscMalloc((n+1)*sizeof(PetscInt),&cnt); > for (i=0;i cnt[i]=irow[i+1]-irow[i]; > } > MatCreateSeqAIJ(PETSC_COMM_SELF,n,m,0,cnt,&A); > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); > // copy over the matrix entries from the inputs > for (i=0;i PetscMalloc(cnt[i]*sizeof(PetscInt),&col); > for (j=0;j col[j]=icol[irow[i]+j-1]-1; > } > for (j=irow[i]-1;j MatSetValues(A,1,&i,cnt[i],col,&ival[irow[i]-1],INSERT_VALUES); > } > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > ierr=PetscPrintf(PETSC_COMM_WORLD,"Matrix A:\n");CHKERRQ(ierr); > MatView(A,PETSC_VIEWER_STDOUT_SELF); > > > /*===================================================================================*/ > > /* Use options from the terminal to create a vector that is type shared > or mpi. */ > ierr = VecCreate(PETSC_COMM_WORLD,&b);CHKERRQ(ierr); /* Vector creation > */ > ierr = VecSetSizes(b,PETSC_DECIDE,n);CHKERRQ(ierr); /* Setting the > vector size */ > ierr = VecSetFromOptions(b);CHKERRQ(ierr); /* Setting the vector type > (shared, mpi etc) */ > /* The second argument is a PETSc scalar value.*/ > ierr = VecSet(b,0);CHKERRQ(ierr); > > /* Reading in the RHS vector. */ > /* Reading in the matrix from the file matrix.txt */ > //fp=fopen("b1.mat","r"); > fp=fopen("rhs.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n; i++) > { > if (fscanf(fp,"%lf", &rhs[i]) != 1) > { > fprintf(stderr, "Failed to read rhs vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > > index=0; > /*Putting x into final form */ > for (i=0; i { > /* Oneinsertion per step. Thats what the 1 in second argument stands > for */ > ierr= VecSetValues(b,1,&index,&rhs[i],INSERT_VALUES);CHKERRQ(ierr); > index=index+1; > } /* The third and fourth arguments are addresses. The fifth argument is > IORA */ > > /* Assembling the vector. */ > ierr= VecAssemblyBegin(b);CHKERRQ(ierr); > ierr=VecAssemblyEnd(b);CHKERRQ(ierr); > > /* Viewing the changed vector. */ > ierr=PetscPrintf(PETSC_COMM_WORLD,"Vector b:\n");CHKERRQ(ierr); > ierr=VecView(b,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > > /*===================================================================================*/ > > /* solve kspslqr system */ > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,m); > VecSetFromOptions(x); > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPLSQR); > KSPSetTolerances(ksp,1.0e-6,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetOperators(ksp, A, A, SAME_PRECONDITIONER); > KSPSetFromOptions(ksp); > KSPSolve(ksp, b, x); > > PetscInt num_iters; > KSPGetIterationNumber(ksp, &num_iters); CHKERRQ(ierr); > > PetscReal rnorm; > ierr = KSPGetResidualNorm(ksp, &rnorm); CHKERRQ(ierr); > > KSPConvergedReason reason; > ierr = KSPGetConvergedReason(ksp, &reason); CHKERRQ(ierr); > > printf("KSPGetIterationNumber %d\n",num_iters); > printf("KSPGetResidualNorm %f\n",rnorm); > printf("KSPConvergedReason %d\n",reason); > > //VecView(x,PETSC_VIEWER_STDOUT_WORLD); > //PetscViewerASCIIOpen(PETSC_COMM_WORLD, "x.data", &viewer); > VecView(x,PETSC_VIEWER_STDOUT_WORLD); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > /* > Destroy any objects created. > */ > ierr=VecDestroy(&b);CHKERRQ(ierr); > ierr=MatDestroy(&A);CHKERRQ(ierr); > ierr=PetscFinalize();CHKERRQ(ierr); > > return 0; > } > > This code read the inputs data and formed the A and b matrix correctly as > I can view them in the output. However, I got ?Invalid argument!? error as > shown below: > > [d3m956 at olympus tutorials]$ ./ss > irow[0]=1 > irow[1]=3 > irow[2]=5 > irow[3]=6 > irow[4]=9 > icol[0]=1 > icol[1]=2 > icol[2]=1 > icol[3]=3 > icol[4]=2 > icol[5]=1 > icol[6]=2 > icol[7]=3 > ival[0]=1.000000 > ival[1]=2.000000 > ival[2]=2.000000 > ival[3]=5.000000 > ival[4]=3.000000 > ival[5]=1.000000 > ival[6]=4.000000 > ival[7]=1.000000 > Matrix A: > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 1) (1, 2) > row 1: (0, 2) (2, 5) > row 2: (1, 3) > row 3: (0, 1) (1, 4) (2, 1) > Vector b: > Vector Object: 1 MPI processes > type: seq > 1 > 2 > 3 > 4 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Must be square matrix, rows 4 columns 3! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ss on a arch-linu named olympus.local by d3m956 Fri Aug > 3 15:34:07 2012 > [0]PETSC ERROR: Libraries linked from > /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 228 in src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 206 in src/ksp/pc/impls/factor/ilu/ilu.c > It is ILU, the default preconditioner, that requires a square matrix. Try -pc_type none Matt > [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c > KSPGetIterationNumber 0 > KSPGetResidualNorm 0.000000 > KSPConvergedReason 0 > Vector Object: 1 MPI processes > type: seq > 0 > 0 > 0 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using diagonal shift to prevent zero pivot > matrix ordering: natural > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=4, cols=3 > total: nonzeros=8, allocated nonzeros=8 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > I don?t understand why the lsqr solver requires a square matrix ?Must be > square matrix, rows 4 columns 3!? > > Can anyone look into the problem for me? > > Thanks a lot! > > Shuangshuang > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Fri Aug 3 19:09:32 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Fri, 3 Aug 2012 17:09:32 -0700 Subject: [petsc-users] error when using ksplsqr for overdetermined system In-Reply-To: Message-ID: That solves my problem. Thank you! Results are updated: [d3m956 at olympus tutorials]$ ./ss -pc_type none irow[0]=1 irow[1]=3 irow[2]=5 irow[3]=6 irow[4]=9 icol[0]=1 icol[1]=2 icol[2]=1 icol[3]=3 icol[4]=2 icol[5]=1 icol[6]=2 icol[7]=3 ival[0]=1.000000 ival[1]=2.000000 ival[2]=2.000000 ival[3]=5.000000 ival[4]=3.000000 ival[5]=1.000000 ival[6]=4.000000 ival[7]=1.000000 Matrix A: Matrix Object: 1 MPI processes type: seqaij row 0: (0, 1) (1, 2) row 1: (0, 2) (2, 5) row 2: (1, 3) row 3: (0, 1) (1, 4) (2, 1) Vector b: Vector Object: 1 MPI processes type: seq 1 2 3 4 KSPGetIterationNumber 3 KSPGetResidualNorm 0.132712 KSPConvergedReason 1 Vector Object: 1 MPI processes type: seq -1.00196 1.0274 0.804305 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=4, cols=3 total: nonzeros=8, allocated nonzeros=8 total number of mallocs used during MatSetValues calls =0 not using I-node routines On 8/3/12 4:35 PM, "Matthew Knepley" wrote: On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang wrote: Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. I have read the following threads regarding the topic: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html I defined a 4*3 matrix A and a vector b in sparse CSR format: irow: 1 3 5 6 9 icol: 1 2 1 3 2 1 2 3 ival: 1 2 2 5 3 1 4 1 rhs: 1 2 3 4 A = 1 2 0 2 0 5 0 3 0 1 4 1 b = 1 2 3 4 I wrote the following code to solve this system: static char help[]="Reading in a matrix\n"; #include #include #include #include "petscvec.h" /* This enables us to use vectors. */ #include "petscmat.h" /* This enables us to use Matrices. It includes the petscvec header file*/ #include "petscksp.h" /* Now we can solve linear systems. Solvers used are KSP. */ int main(int argc, char **argv) { /* Declaration of Matrix A and some vectors x*/ Vec b,x; Mat A; FILE *fp; MPI_Comm comm; PetscInt n=4,m=3,nz=8,index; PetscErrorCode ierr; PetscInt i,j; comm = MPI_COMM_SELF; PetscInt irow[n+1],icol[nz]; PetscScalar ival[nz],rhs[n]; PetscInt *cnt,*col; KSP ksp; /* This part is needed for the help flag supplied at run-time*/ ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); /*===================================================================================*/ fp=fopen("irow.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < n+1; i++) { if (fscanf(fp,"%d", &irow[i]) != 1) { fprintf(stderr, "Failed to read irow vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); fp=fopen("icol.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%d", &icol[i]) != 1) { fprintf(stderr, "Failed to read icol vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); fp=fopen("ival.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%lf", &ival[i]) != 1) { fprintf(stderr, "Failed to read ival vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); /*===================================================================================*/ // determine number of nonzeros per row in the new matrix PetscMalloc((n+1)*sizeof(PetscInt),&cnt); for (i=0;i References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> Message-ID: <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: C_log_summary Type: application/octet-stream Size: 10079 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: G_log_summary Type: application/octet-stream Size: 10430 bytes Desc: not available URL: From zengshixiangze at 163.com Sat Aug 4 00:24:35 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sat, 4 Aug 2012 13:24:35 +0800 (CST) Subject: [petsc-users] The memory of GPU isn't enough for the matrix, can the matrix divide into several parts automatically? Message-ID: <246c4757.17419.138f0179d2b.Coremail.zengshixiangze@163.com> Dear all, When I use GPU to solve the equations, it appears the error: terminate called after throwing an instance of 'thrust::system::detail::bad_alloc' what(): std::bad_alloc: out of memory When the system is smaller, it doesn't appear this error. And I'm sure the memory is enough for the larger system, is it the memory of the GPU that leads to this error? The GPU I use is Nvidia Quadro 4000. The larger system is 3847957x3847957, with 109189295 non-0 elements. And what can we do if the memory of the GPU isn't enough? Thank you! Sincerely, Zeng Xiangze -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From azamat.mametjanov at gmail.com Sat Aug 4 01:40:33 2012 From: azamat.mametjanov at gmail.com (Azamat Mametjanov) Date: Sat, 4 Aug 2012 01:40:33 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> Message-ID: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: > Dear Matt, > > My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. > The size of the system is 2522469 x 2522469, and the number non-0 elements > is 71773925, about 0.000012 of the total. > The output of -log_summary is in the attachment. The G_log_summary is the > output when using GPU, C_log_summary when using CPU. > > Zeng Xiangze > > ? 2012-08-03 22:28:07?"Matthew Knepley" ??? > > On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: > >> Dear all, >> >> When I use the CPU solve the equations, it takes 78 minutes, when I >> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >> paper say when using PETCs with GPU to solve the large sparse matrix >> equations, it can be several times faster? What's the matter? >> > > For all performance questions, we at least need the output of > -log_summary. However, we would also need to know > > - The size and sparsity of your system > > - The CPU and GPU you used (saying anything without knowing this is > impossible) > > Matt > > >> Thank you! >> >> Sincerely, >> Zeng Xiangze >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From azamat.mametjanov at gmail.com Sat Aug 4 01:52:27 2012 From: azamat.mametjanov at gmail.com (Azamat Mametjanov) Date: Sat, 4 Aug 2012 01:52:27 -0500 Subject: [petsc-users] The memory of GPU isn't enough for the matrix, can the matrix divide into several parts automatically? In-Reply-To: <246c4757.17419.138f0179d2b.Coremail.zengshixiangze@163.com> References: <246c4757.17419.138f0179d2b.Coremail.zengshixiangze@163.com> Message-ID: It would probably be best to divide it manually at the moment. Automatic pipelining of a matrix for a given GPU (2GB memory here) can be handled by an autotuner. On Sat, Aug 4, 2012 at 12:24 AM, Xiangze Zeng wrote: > Dear all, > > When I use GPU to solve the equations, it appears the error: > > terminate called after throwing an instance of > 'thrust::system::detail::bad_alloc' > what(): std::bad_alloc: out of memory > > When the system is smaller, it doesn't appear this error. And I'm sure the > memory is enough for the larger system, is it the memory of the GPU that > leads to this error? The GPU I use is Nvidia Quadro 4000. The larger system > is 3847957x3847957, with 109189295 non-0 elements. > > And what can we do if the memory of the GPU isn't enough? > > Thank you! > Sincerely, > Zeng Xiangze > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Sat Aug 4 05:58:17 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sat, 4 Aug 2012 18:58:17 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> Message-ID: <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_log_3 Type: application/octet-stream Size: 9267 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_log_3 Type: application/octet-stream Size: 9619 bytes Desc: not available URL: From dharmareddy84 at gmail.com Sat Aug 4 08:32:06 2012 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Sat, 4 Aug 2012 08:32:06 -0500 Subject: [petsc-users] Installing Petsc Message-ID: Hello, I have a petsc installed in debug mode with following options: Configure Options: --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_rScalar --with-mpi-dir=/opt/mpich2-current --with-scalar-type=real --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 --with-exodusii-dir=/opt/exodusii-4.98 --with-netcdf-dir=/opt/netcdf_v4.1.1 --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 Working directory: /opt/petsc/petsc-3.2-p5 Now i need to build an optimized version of petsc what should be the configure options? Do i need to metion the download options again ? My computer has dual intel Xeon E5420, I am compiling with FORTRAN and C, what should be the Optimization flags? thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 08:36:51 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 08:36:51 -0500 Subject: [petsc-users] The memory of GPU isn't enough for the matrix, can the matrix divide into several parts automatically? In-Reply-To: <246c4757.17419.138f0179d2b.Coremail.zengshixiangze@163.com> References: <246c4757.17419.138f0179d2b.Coremail.zengshixiangze@163.com> Message-ID: On Sat, Aug 4, 2012 at 12:24 AM, Xiangze Zeng wrote: > Dear all, > > When I use GPU to solve the equations, it appears the error: > > terminate called after throwing an instance of > 'thrust::system::detail::bad_alloc' > what(): std::bad_alloc: out of memory > > When the system is smaller, it doesn't appear this error. And I'm sure the > memory is enough for the larger system, is it the memory of the GPU that > leads to this error? The GPU I use is Nvidia Quadro 4000. The larger system > is 3847957x3847957, with 109189295 non-0 elements. > > And what can we do if the memory of the GPU isn't enough? > Run in parallel on more than 1 GPU. Matt > Thank you! > Sincerely, > Zeng Xiangze > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 08:44:02 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 08:44:02 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> Message-ID: On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: > After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 > minutes, a little better than before. The attachment are the output of > -log_summary. > 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt > At 2012-08-04 14:40:33,"Azamat Mametjanov" > wrote: > > What happens if you try to re-run with "--with-debugging=no"? > > On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: > >> Dear Matt, >> >> My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. >> The size of the system is 2522469 x 2522469, and the number non-0 >> elements is 71773925, about 0.000012 of the total. >> The output of -log_summary is in the attachment. The G_log_summary is the >> output when using GPU, C_log_summary when using CPU. >> >> Zeng Xiangze >> >> ? 2012-08-03 22:28:07?"Matthew Knepley" ??? >> >> On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: >> >>> Dear all, >>> >>> When I use the CPU solve the equations, it takes 78 minutes, when I >>> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >>> paper say when using PETCs with GPU to solve the large sparse matrix >>> equations, it can be several times faster? What's the matter? >>> >> >> For all performance questions, we at least need the output of >> -log_summary. However, we would also need to know >> >> - The size and sparsity of your system >> >> - The CPU and GPU you used (saying anything without knowing this is >> impossible) >> >> Matt >> >> >>> Thank you! >>> >>> Sincerely, >>> Zeng Xiangze >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 08:44:41 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 08:44:41 -0500 Subject: [petsc-users] Installing Petsc In-Reply-To: References: Message-ID: On Sat, Aug 4, 2012 at 8:32 AM, Dharmendar Reddy wrote: > Hello, > I have a petsc installed in debug mode with following options: > > Configure Options: --configModules=PETSc.Configure > --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_rScalar > --with-mpi-dir=/opt/mpich2-current --with-scalar-type=real > --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 > --with-exodusii-dir=/opt/exodusii-4.98 --with-netcdf-dir=/opt/netcdf_v4.1.1 > --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 > --download-scalapack=1 --download-blacs=1 > > Working directory: /opt/petsc/petsc-3.2-p5 > > Now i need to build an optimized version of petsc what should be the > configure options? Do i need to metion the download options again ? My > computer has dual intel Xeon E5420, I am compiling with FORTRAN and C, what > should be the Optimization flags? > --with-debugging=no Matt > thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Sat Aug 4 08:50:48 2012 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Sat, 4 Aug 2012 08:50:48 -0500 Subject: [petsc-users] Installing Petsc In-Reply-To: References: Message-ID: Hello Matthew, I Do i need to download the external packages again? I mean, if i give the download options again and add debuggin=no, will the packages be downloaded again or will they be used from the existing downloads in the folder external packages ? On Sat, Aug 4, 2012 at 8:44 AM, Matthew Knepley wrote: > On Sat, Aug 4, 2012 at 8:32 AM, Dharmendar Reddy wrote: > >> Hello, >> I have a petsc installed in debug mode with following options: >> >> Configure Options: --configModules=PETSc.Configure >> --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_rScalar >> --with-mpi-dir=/opt/mpich2-current --with-scalar-type=real >> --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 >> --with-exodusii-dir=/opt/exodusii-4.98 --with-netcdf-dir=/opt/netcdf_v4.1.1 >> --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 >> --download-scalapack=1 --download-blacs=1 >> >> Working directory: /opt/petsc/petsc-3.2-p5 >> >> Now i need to build an optimized version of petsc what should be the >> configure options? Do i need to metion the download options again ? My >> computer has dual intel Xeon E5420, I am compiling with FORTRAN and C, what >> should be the Optimization flags? >> > > --with-debugging=no > > Matt > > >> thanks >> Reddy >> >> -- >> ----------------------------------------------------- >> Dharmendar Reddy Palle >> Graduate Student >> Microelectronics Research center, >> University of Texas at Austin, >> 10100 Burnet Road, Bldg. 160 >> MER 2.608F, TX 78758-4445 >> e-mail: dharmareddy84 at gmail.com >> Phone: +1-512-350-9082 >> United States of America. >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Sat Aug 4 09:06:25 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sat, 4 Aug 2012 22:06:25 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> Message-ID: <669c37ec.1cb2d.138f1f55eb7.Coremail.zengshixiangze@163.com> Sorry about I forgot about the PC type. I'll try again. Thank you so much? Zeng Xiangze ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 09:08:24 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 09:08:24 -0500 Subject: [petsc-users] Installing Petsc In-Reply-To: References: Message-ID: On Sat, Aug 4, 2012 at 8:50 AM, Dharmendar Reddy wrote: > Hello Matthew, > I Do i need to download the external packages > again? I mean, if i give the download options again and add debuggin=no, > will the packages be downloaded again or will they be used from the > existing downloads in the folder external packages ? > It will use the existing packages. Matt > On Sat, Aug 4, 2012 at 8:44 AM, Matthew Knepley wrote: > >> On Sat, Aug 4, 2012 at 8:32 AM, Dharmendar Reddy > > wrote: >> >>> Hello, >>> I have a petsc installed in debug mode with following options: >>> >>> Configure Options: --configModules=PETSc.Configure >>> --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_rScalar >>> --with-mpi-dir=/opt/mpich2-current --with-scalar-type=real >>> --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 >>> --with-exodusii-dir=/opt/exodusii-4.98 --with-netcdf-dir=/opt/netcdf_v4.1.1 >>> --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 >>> --download-scalapack=1 --download-blacs=1 >>> >>> Working directory: /opt/petsc/petsc-3.2-p5 >>> >>> Now i need to build an optimized version of petsc what should be the >>> configure options? Do i need to metion the download options again ? My >>> computer has dual intel Xeon E5420, I am compiling with FORTRAN and C, what >>> should be the Optimization flags? >>> >> >> --with-debugging=no >> >> Matt >> >> >>> thanks >>> Reddy >>> >>> -- >>> ----------------------------------------------------- >>> Dharmendar Reddy Palle >>> Graduate Student >>> Microelectronics Research center, >>> University of Texas at Austin, >>> 10100 Burnet Road, Bldg. 160 >>> MER 2.608F, TX 78758-4445 >>> e-mail: dharmareddy84 at gmail.com >>> Phone: +1-512-350-9082 >>> United States of America. >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Sat Aug 4 09:42:55 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sat, 4 Aug 2012 22:42:55 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> Message-ID: <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 10:04:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 10:04:55 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> Message-ID: On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: > Another error happens when I change the PC type. When I change it to > PCJACOBI, it appears the following error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: Divide by zero! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 > -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by > hongwang Sat Aug 4 22:23:58 2012 > [0]PETSC ERROR: Libraries linked from > /usr/src/petsc/petsc-dev/arch-cuda-double/lib > [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 > [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 > --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 > --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 > --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 > --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 > --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 > --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 > --with-clanguage=c --with-cuda-arch=sm_20 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c > > And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of > memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the > result is not better than that when I don't change the type. > This is breakdown in that algorithm. Try GMRES. Matt > Does it have something to do with the KSP type? Should I look for a suited > KSP type to match the PC type which can work on the GPU? > > ? 2012-08-04 21:44:02?"Matthew Knepley" ??? > > On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: > >> After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 >> minutes, a little better than before. The attachment are the output of >> -log_summary. >> > > 1) Notice how the PCApply takes most of the time, so MatMult is not very > important > > 2) In g_log_3, notice that every time your PC is called, the vector is > pulled from the GPU to the CPU. > This means we do not support that PC on the GPU > > There is a restriction on PCs since not many are coded for the GPU. > Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP > work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. > > Matt > > >> At 2012-08-04 14:40:33,"Azamat Mametjanov" >> wrote: >> >> What happens if you try to re-run with "--with-debugging=no"? >> >> On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: >> >>> Dear Matt, >>> >>> My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. >>> The size of the system is 2522469 x 2522469, and the number non-0 >>> elements is 71773925, about 0.000012 of the total. >>> The output of -log_summary is in the attachment. The G_log_summary is >>> the output when using GPU, C_log_summary when using CPU. >>> >>> Zeng Xiangze >>> >>> ? 2012-08-03 22:28:07?"Matthew Knepley" ??? >>> >>> On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: >>> >>>> Dear all, >>>> >>>> When I use the CPU solve the equations, it takes 78 minutes, when I >>>> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >>>> paper say when using PETCs with GPU to solve the large sparse matrix >>>> equations, it can be several times faster? What's the matter? >>>> >>> >>> For all performance questions, we at least need the output of >>> -log_summary. However, we would also need to know >>> >>> - The size and sparsity of your system >>> >>> - The CPU and GPU you used (saying anything without knowing this is >>> impossible) >>> >>> Matt >>> >>> >>>> Thank you! >>>> >>>> Sincerely, >>>> Zeng Xiangze >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> >> >> >> -- >> Mailbox 379, School of Physics >> Shandong University >> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Sat Aug 4 11:01:11 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sun, 5 Aug 2012 00:01:11 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> Message-ID: <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the manual, is it KSPBCGS? ? 2012-08-04 23:04:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. This is breakdown in that algorithm. Try GMRES. Matt Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 4 11:17:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Aug 2012 11:17:05 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> Message-ID: On Sat, Aug 4, 2012 at 11:01 AM, Xiangze Zeng wrote: > JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. > When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In > http://www.mcs.anl.gov/petsc/features/gpus.html, I see " All of the > Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS > in the manual, is it KSPBCGS? > No. Matt > ? 2012-08-04 23:04:55?"Matthew Knepley" ??? > > On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: > >> Another error happens when I change the PC type. When I change it to >> PCJACOBI, it appears the following error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Petsc has generated inconsistent data! >> [0]PETSC ERROR: Divide by zero! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Development HG revision: >> d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 >> -0500 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by >> hongwang Sat Aug 4 22:23:58 2012 >> [0]PETSC ERROR: Libraries linked from >> /usr/src/petsc/petsc-dev/arch-cuda-double/lib >> [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 >> [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 >> --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 >> --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 >> --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 >> --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 >> --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 >> --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 >> --with-clanguage=c --with-cuda-arch=sm_20 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c >> >> And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of >> memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the >> result is not better than that when I don't change the type. >> > > This is breakdown in that algorithm. Try GMRES. > > Matt > > >> Does it have something to do with the KSP type? Should I look for a >> suited KSP type to match the PC type which can work on the GPU? >> >> ? 2012-08-04 21:44:02?"Matthew Knepley" ??? >> >> On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: >> >>> After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 >>> minutes, a little better than before. The attachment are the output of >>> -log_summary. >>> >> >> 1) Notice how the PCApply takes most of the time, so MatMult is not very >> important >> >> 2) In g_log_3, notice that every time your PC is called, the vector is >> pulled from the GPU to the CPU. >> This means we do not support that PC on the GPU >> >> There is a restriction on PCs since not many are coded for the GPU. >> Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP >> work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. >> >> Matt >> >> >>> At 2012-08-04 14:40:33,"Azamat Mametjanov" >>> wrote: >>> >>> What happens if you try to re-run with "--with-debugging=no"? >>> >>> On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: >>> >>>> Dear Matt, >>>> >>>> My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. >>>> The size of the system is 2522469 x 2522469, and the number non-0 >>>> elements is 71773925, about 0.000012 of the total. >>>> The output of -log_summary is in the attachment. The G_log_summary is >>>> the output when using GPU, C_log_summary when using CPU. >>>> >>>> Zeng Xiangze >>>> >>>> ? 2012-08-03 22:28:07?"Matthew Knepley" ??? >>>> >>>> On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: >>>> >>>>> Dear all, >>>>> >>>>> When I use the CPU solve the equations, it takes 78 minutes, when I >>>>> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >>>>> paper say when using PETCs with GPU to solve the large sparse matrix >>>>> equations, it can be several times faster? What's the matter? >>>>> >>>> >>>> For all performance questions, we at least need the output of >>>> -log_summary. However, we would also need to know >>>> >>>> - The size and sparsity of your system >>>> >>>> - The CPU and GPU you used (saying anything without knowing this is >>>> impossible) >>>> >>>> Matt >>>> >>>> >>>>> Thank you! >>>>> >>>>> Sincerely, >>>>> Zeng Xiangze >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Mailbox 379, School of Physics >>> Shandong University >>> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> -- >> Mailbox 379, School of Physics >> Shandong University >> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From petsc-maint at mcs.anl.gov Sat Aug 4 12:28:03 2012 From: petsc-maint at mcs.anl.gov (Matthew Knepley) Date: Sat, 4 Aug 2012 12:28:03 -0500 Subject: [petsc-users] [petsc-maint #127161] Re: Installing Petsc In-Reply-To: References: Message-ID: On Sat, Aug 4, 2012 at 12:11 PM, Dharmendar Reddy wrote: > Hello Matthew, > I installed petsc with the following options. > configOpts= --with-cc=mpicc \ > --with-fc=mpif90 \ > --with-debugging=0 \ > --with-fortran-interfaces=1 \ > --with-fortran-datatypes=0 \ > --with-petsc-arch=${PETSC_ARCH} \ > --with-petsc-dir=${PETSC_DIR} \ > --with-mpi-dir=/opt/mpich2-current \ > --with-scalar-type=real \ > --with-blas-lapack-dir=/opt/intel/lib/intel64 \ > --download-umfpack=1 \ > --with-exodusii-dir=/opt/exodusii-4.98 \ > --with-netcdf-dir=/opt/netcdf_v4.1.1 \ > --download-superlu_dist=1 \ > --download-parmetis=1 \ > --download-mumps=1 \ > --download-scalapack=1 \ > --download-blacs=1 > > I have attached the configure.log and make.log > > I get the following error when i compile my code (there are lot of errors > complaining about TS). > > /opt/petsc/petsc-3.2-p5/include/finclude/ftn-auto/petscts.h90(2): error > #5082: Syntax error > , found IDENTIFIER 'TS' when expecting one of: ( % [ : . = => > TS ts ! TS > Send the FULL compile output to petsc-maint. Matt > thanks > Reddy > On Sat, Aug 4, 2012 at 9:08 AM, Matthew Knepley wrote: > > > On Sat, Aug 4, 2012 at 8:50 AM, Dharmendar Reddy < > dharmareddy84 at gmail.com>wrote: > > > >> Hello Matthew, > >> I Do i need to download the external packages > >> again? I mean, if i give the download options again and add debuggin=no, > >> will the packages be downloaded again or will they be used from the > >> existing downloads in the folder external packages ? > >> > > > > It will use the existing packages. > > > > Matt > > > > > >> On Sat, Aug 4, 2012 at 8:44 AM, Matthew Knepley >wrote: > >> > >>> On Sat, Aug 4, 2012 at 8:32 AM, Dharmendar Reddy < > >>> dharmareddy84 at gmail.com> wrote: > >>> > >>>> Hello, > >>>> I have a petsc installed in debug mode with following options: > >>>> > >>>> Configure Options: --configModules=PETSc.Configure > >>>> --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_rScalar > >>>> --with-mpi-dir=/opt/mpich2-current --with-scalar-type=real > >>>> --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 > >>>> --with-exodusii-dir=/opt/exodusii-4.98 > --with-netcdf-dir=/opt/netcdf_v4.1.1 > >>>> --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 > >>>> --download-scalapack=1 --download-blacs=1 > >>>> > >>>> Working directory: /opt/petsc/petsc-3.2-p5 > >>>> > >>>> Now i need to build an optimized version of petsc what should be the > >>>> configure options? Do i need to metion the download options again ? My > >>>> computer has dual intel Xeon E5420, I am compiling with FORTRAN and > C, what > >>>> should be the Optimization flags? > >>>> > >>> > >>> --with-debugging=no > >>> > >>> Matt > >>> > >>> > >>>> thanks > >>>> Reddy > >>>> > >>>> -- > >>>> ----------------------------------------------------- > >>>> Dharmendar Reddy Palle > >>>> Graduate Student > >>>> Microelectronics Research center, > >>>> University of Texas at Austin, > >>>> 10100 Burnet Road, Bldg. 160 > >>>> MER 2.608F, TX 78758-4445 > >>>> e-mail: dharmareddy84 at gmail.com > >>>> Phone: +1-512-350-9082 > >>>> United States of America. > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >> > >> > >> > >> -- > >> ----------------------------------------------------- > >> Dharmendar Reddy Palle > >> Graduate Student > >> Microelectronics Research center, > >> University of Texas at Austin, > >> 10100 Burnet Road, Bldg. 160 > >> MER 2.608F, TX 78758-4445 > >> e-mail: dharmareddy84 at gmail.com > >> Phone: +1-512-350-9082 > >> United States of America. > >> > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. > > -- Norbert Wiener > > > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Sat Aug 4 22:56:13 2012 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Sat, 4 Aug 2012 22:56:13 -0500 Subject: [petsc-users] gmg vs amg Message-ID: Hi All, Is it almost always the case that for structured mesh, geometric multigrid would be faster than algebraic multigrid solver/preconditioner? Basically I want to know If I had a choice between pfmg , gamg and boomeramg for structured grid what should I choose? Thanks, -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Aug 4 22:59:42 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Aug 2012 22:59:42 -0500 Subject: [petsc-users] gmg vs amg In-Reply-To: References: Message-ID: On Aug 4, 2012, at 10:56 PM, Amneet Bhalla wrote: > Hi All, > > Is it almost always the case that for structured mesh, geometric multigrid would be faster than algebraic multigrid solver/preconditioner? Not always, if it has highly varying anisotropic coefficients for example > Basically I want to know If I had a choice between pfmg , gamg and boomeramg for structured grid what should I choose? I would try the algebraic multigrid since it is trivial to try and if that gives nice multigrid convergence, end of story, no need to spend time providing extra geometric information. Barry > > Thanks, > > -- > Amneet > > > > From zengshixiangze at 163.com Sat Aug 4 23:23:08 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sun, 5 Aug 2012 12:23:08 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> Message-ID: <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> When I change the PC type to JACOBI, the KSP type to BICG, although the computational speed both in the GPU and CPU are higher than that when I use SOR+BCGS, the computational work in the GPU doesn't seem much more efficient, the speed only 20% higher. Is there any proposal? The attachments are the output of the log_summary. Thank you! Zeng Xiangze At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the manual, is it KSPBCGS? ? 2012-08-04 23:04:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. This is breakdown in that algorithm. Try GMRES. Matt Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_log_4 Type: application/octet-stream Size: 9242 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_log_4 Type: application/octet-stream Size: 9594 bytes Desc: not available URL: From knepley at gmail.com Sun Aug 5 07:27:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 5 Aug 2012 07:27:55 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> Message-ID: On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng wrote: > When I change the PC type to JACOBI, the KSP type to BICG, although the > computational speed both in the GPU and CPU are higher than that when I use > SOR+BCGS, the computational work in the GPU doesn't seem much more > efficient, the speed only 20% higher. Is there any proposal? The > attachments are the output of the log_summary. > You also have to look at the log_summary: VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1) I said to use GMRES for a reason. Listen to me. BiCG uses the transpose, which right now confuses the results 2) Look at the copies to/from the GPU. You should not be copying the vector 4000 times. Start simple until you understand everything about how the code is running. Use -pc_type none -ksp_type gmres and see if you can understand the results. Then try different KSP and PC. Trying everything at once does not help anyone, and it is not science. Matt > Thank you! > > Zeng Xiangze > > At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: > > JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. > When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In > http://www.mcs.anl.gov/petsc/features/gpus.html, I see " All of the > Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS > in the manual, is it KSPBCGS? > > ? 2012-08-04 23:04:55?"Matthew Knepley" ??? > > On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: > >> Another error happens when I change the PC type. When I change it to >> PCJACOBI, it appears the following error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Petsc has generated inconsistent data! >> [0]PETSC ERROR: Divide by zero! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Development HG revision: >> d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 >> -0500 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by >> hongwang Sat Aug 4 22:23:58 2012 >> [0]PETSC ERROR: Libraries linked from >> /usr/src/petsc/petsc-dev/arch-cuda-double/lib >> [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 >> [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 >> --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 >> --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 >> --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 >> --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 >> --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 >> --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 >> --with-clanguage=c --with-cuda-arch=sm_20 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c >> >> And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of >> memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the >> result is not better than that when I don't change the type. >> > > This is breakdown in that algorithm. Try GMRES. > > Matt > > >> Does it have something to do with the KSP type? Should I look for a >> suited KSP type to match the PC type which can work on the GPU? >> >> ? 2012-08-04 21:44:02?"Matthew Knepley" ??? >> >> On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: >> >>> After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 >>> minutes, a little better than before. The attachment are the output of >>> -log_summary. >>> >> >> 1) Notice how the PCApply takes most of the time, so MatMult is not very >> important >> >> 2) In g_log_3, notice that every time your PC is called, the vector is >> pulled from the GPU to the CPU. >> This means we do not support that PC on the GPU >> >> There is a restriction on PCs since not many are coded for the GPU. >> Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP >> work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. >> >> Matt >> >> >>> At 2012-08-04 14:40:33,"Azamat Mametjanov" >>> wrote: >>> >>> What happens if you try to re-run with "--with-debugging=no"? >>> >>> On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: >>> >>>> Dear Matt, >>>> >>>> My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. >>>> The size of the system is 2522469 x 2522469, and the number non-0 >>>> elements is 71773925, about 0.000012 of the total. >>>> The output of -log_summary is in the attachment. The G_log_summary is >>>> the output when using GPU, C_log_summary when using CPU. >>>> >>>> Zeng Xiangze >>>> >>>> ? 2012-08-03 22:28:07?"Matthew Knepley" ??? >>>> >>>> On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: >>>> >>>>> Dear all, >>>>> >>>>> When I use the CPU solve the equations, it takes 78 minutes, when I >>>>> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >>>>> paper say when using PETCs with GPU to solve the large sparse matrix >>>>> equations, it can be several times faster? What's the matter? >>>>> >>>> >>>> For all performance questions, we at least need the output of >>>> -log_summary. However, we would also need to know >>>> >>>> - The size and sparsity of your system >>>> >>>> - The CPU and GPU you used (saying anything without knowing this is >>>> impossible) >>>> >>>> Matt >>>> >>>> >>>>> Thank you! >>>>> >>>>> Sincerely, >>>>> Zeng Xiangze >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Mailbox 379, School of Physics >>> Shandong University >>> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> -- >> Mailbox 379, School of Physics >> Shandong University >> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Sun Aug 5 10:24:08 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Sun, 5 Aug 2012 23:24:08 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> Message-ID: Dear Matt, Thank you for your suggestion. I'm learning to use the GPU effectively step by step. I think it's useful for the novice if there is a manual about using PETSc with CUDA. Each iteration is done, the VEC will be copied to the host to evaluate the stopping condition, is it right? Sincerely, Zeng Xiangze ? 2012-08-05 20:27:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng wrote: When I change the PC type to JACOBI, the KSP type to BICG, although the computational speed both in the GPU and CPU are higher than that when I use SOR+BCGS, the computational work in the GPU doesn't seem much more efficient, the speed only 20% higher. Is there any proposal? The attachments are the output of the log_summary. You also have to look at the log_summary: VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1) I said to use GMRES for a reason. Listen to me. BiCG uses the transpose, which right now confuses the results 2) Look at the copies to/from the GPU. You should not be copying the vector 4000 times. Start simple until you understand everything about how the code is running. Use -pc_type none -ksp_type gmres and see if you can understand the results. Then try different KSP and PC. Trying everything at once does not help anyone, and it is not science. Matt Thank you! Zeng Xiangze At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the manual, is it KSPBCGS? ? 2012-08-04 23:04:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. This is breakdown in that algorithm. Try GMRES. Matt Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pinna.2006 at gmail.com Sun Aug 5 10:27:44 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Sun, 5 Aug 2012 17:27:44 +0200 Subject: [petsc-users] Configure error Message-ID: Dear all, I am new to petsc, and I've been struggling with it for the last days. I want to install it on Windows 7, and first I tried to install the Microsoft MPI and then compiling the MPICH. They both seems ok but I am not able to compile Petsc with any of them. I am mostly interested to MPICH (for portability). I used this script to configure ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 --download-f2cblaslapack=1 ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 and I've got this error, UNABLE to EXECUTE BINARIES for config/configure.py --------------------------------------------------------------------------------------- [Errno 16] Device or resource busy: 'conftest.exe' This error seems to happen pretty randomly at differente places in the script. Any idea to solve this problem? Thank you all Fabio From bsmith at mcs.anl.gov Sun Aug 5 11:47:18 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 5 Aug 2012 11:47:18 -0500 Subject: [petsc-users] Configure error In-Reply-To: References: Message-ID: On Aug 5, 2012, at 10:27 AM, Fabio wrote: > Dear all, > I am new to petsc, and I've been struggling with it for the last days. > > I want to install it on Windows 7, and first I tried to install the > Microsoft MPI and then compiling the MPICH. They both seems ok but I > am not able to compile Petsc with any of them. > I am mostly interested to MPICH (for portability). > I used this script to configure > > ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 > --download-f2cblaslapack=1 > ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 > > and I've got this error, > > UNABLE to EXECUTE BINARIES for config/configure.py > --------------------------------------------------------------------------------------- > [Errno 16] Device or resource busy: 'conftest.exe' > > This error seems to happen pretty randomly at differente places in the script. > Any idea to solve this problem? > Thank you all This is due to bad interactions between cygwin and windows. First try rebooting. Next have as little other load on the machine as possible. Barry > > Fabio From balay at mcs.anl.gov Sun Aug 5 11:48:43 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 5 Aug 2012 11:48:43 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: Message-ID: On Sun, 5 Aug 2012, Fabio wrote: > Dear all, > I am new to petsc, and I've been struggling with it for the last days. > > I want to install it on Windows 7, and first I tried to install the > Microsoft MPI and then compiling the MPICH. They both seems ok but I > am not able to compile Petsc with any of them. > I am mostly interested to MPICH (for portability). > I used this script to configure > > ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 > --download-f2cblaslapack=1 > ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? [MS MPI is for use with MS compiler only. MPICH binary install is for MS compilers, and MPICH source install is for cygwin gcc/g++] > > and I've got this error, > > UNABLE to EXECUTE BINARIES for config/configure.py > --------------------------------------------------------------------------------------- > [Errno 16] Device or resource busy: 'conftest.exe' > > This error seems to happen pretty randomly at differente places in the script. > Any idea to solve this problem? Its bad file locks on windows. The only suggestion we have is reboot. For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be that issue. If you still have issues - send the complete log [and logfile] to petsc-maint. Satish > Thank you all > > Fabio > From knepley at gmail.com Sun Aug 5 14:18:58 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 5 Aug 2012 14:18:58 -0500 Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> Message-ID: On Sun, Aug 5, 2012 at 10:24 AM, Xiangze Zeng wrote: > Dear Matt, > > Thank you for your suggestion. I'm learning to use the GPU effectively > step by step. I think it's useful for the novice if there is a manual about > using PETSc with CUDA. > Each iteration is done, the VEC will be copied to the host to evaluate the > stopping condition, is it right? > No, if that was true, we would have given up long ago. My guess is that some of your Vecs are not the correct type. Can you look at ex5 suing -dm_vec_type veccusp -dm_mat_type mataijcusp and mail petsc-maint at mcs.anl.gov? Matt > Sincerely, > Zeng Xiangze > > ? 2012-08-05 20:27:55?"Matthew Knepley" ??? > > On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng wrote: > >> When I change the PC type to JACOBI, the KSP type to BICG, although the >> computational speed both in the GPU and CPU are higher than that when I use >> SOR+BCGS, the computational work in the GPU doesn't seem much more >> efficient, the speed only 20% higher. Is there any proposal? The >> attachments are the output of the log_summary. >> > > You also have to look at the log_summary: > > VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 > MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 1) I said to use GMRES for a reason. Listen to me. BiCG uses the > transpose, which right now confuses the results > > 2) Look at the copies to/from the GPU. You should not be copying the > vector 4000 times. Start simple until you understand > everything about how the code is running. Use -pc_type none -ksp_type > gmres and see if you can understand the results. > Then try different KSP and PC. Trying everything at once does not help > anyone, and it is not science. > > Matt > > >> Thank you! >> >> Zeng Xiangze >> >> At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: >> >> JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. >> When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In >> http://www.mcs.anl.gov/petsc/features/gpus.html, I see " All of the >> Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS >> in the manual, is it KSPBCGS? >> >> ? 2012-08-04 23:04:55?"Matthew Knepley" ??? >> >> On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: >> >>> Another error happens when I change the PC type. When I change it to >>> PCJACOBI, it appears the following error message: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Petsc has generated inconsistent data! >>> [0]PETSC ERROR: Divide by zero! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: >>> d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 >>> -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by >>> hongwang Sat Aug 4 22:23:58 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /usr/src/petsc/petsc-dev/arch-cuda-double/lib >>> [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 >>> [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 >>> --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 >>> --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 >>> --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 >>> --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 >>> --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 >>> --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 >>> --with-clanguage=c --with-cuda-arch=sm_20 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c >>> >>> And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of >>> memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the >>> result is not better than that when I don't change the type. >>> >> >> This is breakdown in that algorithm. Try GMRES. >> >> Matt >> >> >>> Does it have something to do with the KSP type? Should I look for a >>> suited KSP type to match the PC type which can work on the GPU? >>> >>> ? 2012-08-04 21:44:02?"Matthew Knepley" ??? >>> >>> On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: >>> >>>> After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 >>>> minutes, a little better than before. The attachment are the output of >>>> -log_summary. >>>> >>> >>> 1) Notice how the PCApply takes most of the time, so MatMult is not very >>> important >>> >>> 2) In g_log_3, notice that every time your PC is called, the vector is >>> pulled from the GPU to the CPU. >>> This means we do not support that PC on the GPU >>> >>> There is a restriction on PCs since not many are coded for the GPU. >>> Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP >>> work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. >>> >>> Matt >>> >>> >>>> At 2012-08-04 14:40:33,"Azamat Mametjanov" >>>> wrote: >>>> >>>> What happens if you try to re-run with "--with-debugging=no"? >>>> >>>> On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: >>>> >>>>> Dear Matt, >>>>> >>>>> My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. >>>>> The size of the system is 2522469 x 2522469, and the number non-0 >>>>> elements is 71773925, about 0.000012 of the total. >>>>> The output of -log_summary is in the attachment. The G_log_summary is >>>>> the output when using GPU, C_log_summary when using CPU. >>>>> >>>>> Zeng Xiangze >>>>> >>>>> ? 2012-08-03 22:28:07?"Matthew Knepley" ??? >>>>> >>>>> On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> When I use the CPU solve the equations, it takes 78 minutes, when I >>>>>> change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some >>>>>> paper say when using PETCs with GPU to solve the large sparse matrix >>>>>> equations, it can be several times faster? What's the matter? >>>>>> >>>>> >>>>> For all performance questions, we at least need the output of >>>>> -log_summary. However, we would also need to know >>>>> >>>>> - The size and sparsity of your system >>>>> >>>>> - The CPU and GPU you used (saying anything without knowing this is >>>>> impossible) >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thank you! >>>>>> >>>>>> Sincerely, >>>>>> Zeng Xiangze >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Mailbox 379, School of Physics >>>> Shandong University >>>> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> -- >>> Mailbox 379, School of Physics >>> Shandong University >>> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> -- >> Mailbox 379, School of Physics >> Shandong University >> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >> >> >> >> >> -- >> Mailbox 379, School of Physics >> Shandong University >> 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Sun Aug 5 15:49:03 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Sun, 5 Aug 2012 13:49:03 -0700 Subject: [petsc-users] error when using ksplsqr for overdetermined system In-Reply-To: Message-ID: Hi, Mat, I replaced the inputs with a larger system (the A matrix has 27090 rows and 14959 columns). With the -pc_type none set up as the runtime option, the system can be solved correctly, but the new problem is the solving time is very slow, taking about 75 seconds to run KSPSolve for the lsqr problem. Is there any faster preconditioning type we can use here instead of "none"? [d3m956 at olympus tutorials]$ mpiexec -n 1 ss -pc_type none Pre KSPSolve Elapsed Time: 0.195198 KSPSolve Elapsed Time: 74.788414 KSPGetIterationNumber 10000 KSPGetResidualNorm 17921.356837 KSPConvergedReason -3 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=27090, cols=14959 total: nonzeros=70903, allocated nonzeros=70903 total number of mallocs used during MatSetValues calls =0 not using I-node routines Overall Elapsed Time: 75.155196 Another question is how to run lsqr in parallel. I tried to use 4 processors for example, but it didn't help to speed up. [d3m956 at olympus tutorials]$ mpiexec -n 4 ss -pc_type none Pre KSPSolve Elapsed Time: 0.177819 Pre KSPSolve Elapsed Time: 0.180807 Pre KSPSolve Elapsed Time: 0.179783 Pre KSPSolve Elapsed Time: 0.181975 KSPSolve Elapsed Time: 73.107033 KSPGetIterationNumber 10000 KSPGetResidualNorm 17921.356837 KSPConvergedReason -3 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=27090, cols=14959 total: nonzeros=70903, allocated nonzeros=70903 total number of mallocs used during MatSetValues calls =0 not using I-node routines Overall Elapsed Time: 73.455209 KSPSolve Elapsed Time: 73.425921 KSPGetIterationNumber 10000 KSPGetResidualNorm 17921.356837 KSPConvergedReason -3 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=27090, cols=14959 total: nonzeros=70903, allocated nonzeros=70903 total number of mallocs used during MatSetValues calls =0 not using I-node routines Overall Elapsed Time: 73.771948 KSPSolve Elapsed Time: 76.629414 KSPGetIterationNumber 10000 KSPGetResidualNorm 17921.356837 KSPConvergedReason -3 KSPSolve Elapsed Time: 76.698510 KSPGetIterationNumber 10000 KSPGetResidualNorm 17921.356837 KSPConvergedReason -3 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=27090, cols=14959 total: nonzeros=70903, allocated nonzeros=70903 total number of mallocs used during MatSetValues calls =0 not using I-node routines Overall Elapsed Time: 77.174527 KSP Object: 1 MPI processes type: lsqr maximum iterations=10000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=27090, cols=14959 total: nonzeros=70903, allocated nonzeros=70903 total number of mallocs used during MatSetValues calls =0 not using I-node routines Overall Elapsed Time: 77.247951 Is there anything I need to do in the code to parallelize the ksp lsqr solver other than specify the mpiexec -n option in the command line? Thanks, Shuangshuang On 8/3/12 4:35 PM, "Matthew Knepley" wrote: On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang wrote: Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. I have read the following threads regarding the topic: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html I defined a 4*3 matrix A and a vector b in sparse CSR format: irow: 1 3 5 6 9 icol: 1 2 1 3 2 1 2 3 ival: 1 2 2 5 3 1 4 1 rhs: 1 2 3 4 A = 1 2 0 2 0 5 0 3 0 1 4 1 b = 1 2 3 4 I wrote the following code to solve this system: static char help[]="Reading in a matrix\n"; #include #include #include #include "petscvec.h" /* This enables us to use vectors. */ #include "petscmat.h" /* This enables us to use Matrices. It includes the petscvec header file*/ #include "petscksp.h" /* Now we can solve linear systems. Solvers used are KSP. */ int main(int argc, char **argv) { /* Declaration of Matrix A and some vectors x*/ Vec b,x; Mat A; FILE *fp; MPI_Comm comm; PetscInt n=4,m=3,nz=8,index; PetscErrorCode ierr; PetscInt i,j; comm = MPI_COMM_SELF; PetscInt irow[n+1],icol[nz]; PetscScalar ival[nz],rhs[n]; PetscInt *cnt,*col; KSP ksp; /* This part is needed for the help flag supplied at run-time*/ ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); /*===================================================================================*/ fp=fopen("irow.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < n+1; i++) { if (fscanf(fp,"%d", &irow[i]) != 1) { fprintf(stderr, "Failed to read irow vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); fp=fopen("icol.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%d", &icol[i]) != 1) { fprintf(stderr, "Failed to read icol vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); fp=fopen("ival.txt","r"); if (fp==NULL) { fprintf(stderr, "Cannot open file\n"); exit(1); } for (i = 0; i < nz; i++) { if (fscanf(fp,"%lf", &ival[i]) != 1) { fprintf(stderr, "Failed to read ival vector[%d]\n", i); exit(1); } } fclose(fp); for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); /*===================================================================================*/ // determine number of nonzeros per row in the new matrix PetscMalloc((n+1)*sizeof(PetscInt),&cnt); for (i=0;i References: Message-ID: On Sun, Aug 5, 2012 at 3:49 PM, Jin, Shuangshuang wrote: > Hi, Mat, I replaced the inputs with a larger system (the A matrix has > 27090 rows and 14959 columns). With the -pc_type none set up as the runtime > option, the system can be solved correctly, but the new problem is the > solving time is very slow, taking about 75 seconds to run KSPSolve for the > lsqr problem. Is there any faster preconditioning type we can use here > instead of "none"? > I have no idea what system you are solving. I suggest consulting the literature first. If you know what preconditioner you want, we can try to help. About parallelism, http://www.mcs.anl.gov/petsc/documentation/faq.html#computers Matt > [d3m956 at olympus tutorials]$ mpiexec -n 1 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.195198 > KSPSolve Elapsed Time: 74.788414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 75.155196 > > Another question is how to run lsqr in parallel. I tried to use 4 > processors for example, but it didn't help to speed up. > > [d3m956 at olympus tutorials]$ mpiexec -n 4 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.177819 > Pre KSPSolve Elapsed Time: 0.180807 > Pre KSPSolve Elapsed Time: 0.179783 > Pre KSPSolve Elapsed Time: 0.181975 > KSPSolve Elapsed Time: 73.107033 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.455209 > KSPSolve Elapsed Time: 73.425921 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.771948 > KSPSolve Elapsed Time: 76.629414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSPSolve Elapsed Time: 76.698510 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.174527 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.247951 > > Is there anything I need to do in the code to parallelize the ksp lsqr > solver other than specify the mpiexec -n option in the > command line? > > Thanks, > Shuangshuang > > > On 8/3/12 4:35 PM, "Matthew Knepley" wrote: > > On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang < > Shuangshuang.Jin at pnnl.gov> wrote: > Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. > I have read the following threads regarding the topic: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html < > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html> > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html < > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html> > > I defined a 4*3 matrix A and a vector b in sparse CSR format: > irow: 1 3 5 6 9 > icol: 1 2 1 3 2 1 2 3 > ival: 1 2 2 5 3 1 4 1 > rhs: 1 2 3 4 > > A = 1 2 0 > 2 0 5 > 0 3 0 > 1 4 1 > b = 1 > 2 > 3 > 4 > > I wrote the following code to solve this system: > > static char help[]="Reading in a matrix\n"; > > #include > #include > #include > #include "petscvec.h" /* This enables us to use vectors. */ > #include "petscmat.h" /* This enables us to use Matrices. It includes the > petscvec header file*/ > #include "petscksp.h" /* Now we can solve linear systems. Solvers used are > KSP. */ > > int main(int argc, char **argv) > { > /* Declaration of Matrix A and some vectors x*/ > Vec b,x; > Mat A; > FILE *fp; > > MPI_Comm comm; > PetscInt n=4,m=3,nz=8,index; > PetscErrorCode ierr; > PetscInt i,j; > comm = MPI_COMM_SELF; > PetscInt irow[n+1],icol[nz]; > PetscScalar ival[nz],rhs[n]; > PetscInt *cnt,*col; > > KSP ksp; > > /* This part is needed for the help flag supplied at run-time*/ > ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); > ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); > > > /*===================================================================================*/ > > fp=fopen("irow.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n+1; i++) > { > if (fscanf(fp,"%d", &irow[i]) != 1) > { > fprintf(stderr, "Failed to read irow vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); > > fp=fopen("icol.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%d", &icol[i]) != 1) > { > fprintf(stderr, "Failed to read icol vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); > > fp=fopen("ival.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%lf", &ival[i]) != 1) > { > fprintf(stderr, "Failed to read ival vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); > > > /*===================================================================================*/ > > // determine number of nonzeros per row in the new matrix > PetscMalloc((n+1)*sizeof(PetscInt),&cnt); > for (i=0;i cnt[i]=irow[i+1]-irow[i]; > } > MatCreateSeqAIJ(PETSC_COMM_SELF,n,m,0,cnt,&A); > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); > // copy over the matrix entries from the inputs > for (i=0;i PetscMalloc(cnt[i]*sizeof(PetscInt),&col); > for (j=0;j col[j]=icol[irow[i]+j-1]-1; > } > for (j=irow[i]-1;j MatSetValues(A,1,&i,cnt[i],col,&ival[irow[i]-1],INSERT_VALUES); > } > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > ierr=PetscPrintf(PETSC_COMM_WORLD,"Matrix A:\n");CHKERRQ(ierr); > MatView(A,PETSC_VIEWER_STDOUT_SELF); > > > /*===================================================================================*/ > > /* Use options from the terminal to create a vector that is type shared > or mpi. */ > ierr = VecCreate(PETSC_COMM_WORLD,&b);CHKERRQ(ierr); /* Vector creation > */ > ierr = VecSetSizes(b,PETSC_DECIDE,n);CHKERRQ(ierr); /* Setting the > vector size */ > ierr = VecSetFromOptions(b);CHKERRQ(ierr); /* Setting the vector type > (shared, mpi etc) */ > /* The second argument is a PETSc scalar value.*/ > ierr = VecSet(b,0);CHKERRQ(ierr); > > /* Reading in the RHS vector. */ > /* Reading in the matrix from the file matrix.txt */ > //fp=fopen("b1.mat","r"); > fp=fopen("rhs.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n; i++) > { > if (fscanf(fp,"%lf", &rhs[i]) != 1) > { > fprintf(stderr, "Failed to read rhs vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > > index=0; > /*Putting x into final form */ > for (i=0; i { > /* Oneinsertion per step. Thats what the 1 in second argument stands > for */ > ierr= VecSetValues(b,1,&index,&rhs[i],INSERT_VALUES);CHKERRQ(ierr); > index=index+1; > } /* The third and fourth arguments are addresses. The fifth argument is > IORA */ > > /* Assembling the vector. */ > ierr= VecAssemblyBegin(b);CHKERRQ(ierr); > ierr=VecAssemblyEnd(b);CHKERRQ(ierr); > > /* Viewing the changed vector. */ > ierr=PetscPrintf(PETSC_COMM_WORLD,"Vector b:\n");CHKERRQ(ierr); > ierr=VecView(b,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > > /*===================================================================================*/ > > /* solve kspslqr system */ > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,m); > VecSetFromOptions(x); > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPLSQR); > KSPSetTolerances(ksp,1.0e-6,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetOperators(ksp, A, A, SAME_PRECONDITIONER); > KSPSetFromOptions(ksp); > KSPSolve(ksp, b, x); > > PetscInt num_iters; > KSPGetIterationNumber(ksp, &num_iters); CHKERRQ(ierr); > > PetscReal rnorm; > ierr = KSPGetResidualNorm(ksp, &rnorm); CHKERRQ(ierr); > > KSPConvergedReason reason; > ierr = KSPGetConvergedReason(ksp, &reason); CHKERRQ(ierr); > > printf("KSPGetIterationNumber %d\n",num_iters); > printf("KSPGetResidualNorm %f\n",rnorm); > printf("KSPConvergedReason %d\n",reason); > > //VecView(x,PETSC_VIEWER_STDOUT_WORLD); > //PetscViewerASCIIOpen(PETSC_COMM_WORLD, "x.data", &viewer); > VecView(x,PETSC_VIEWER_STDOUT_WORLD); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > /* > Destroy any objects created. > */ > ierr=VecDestroy(&b);CHKERRQ(ierr); > ierr=MatDestroy(&A);CHKERRQ(ierr); > ierr=PetscFinalize();CHKERRQ(ierr); > > return 0; > } > > This code read the inputs data and formed the A and b matrix correctly as > I can view them in the output. However, I got "Invalid argument!" error as > shown below: > > [d3m956 at olympus tutorials]$ ./ss > irow[0]=1 > irow[1]=3 > irow[2]=5 > irow[3]=6 > irow[4]=9 > icol[0]=1 > icol[1]=2 > icol[2]=1 > icol[3]=3 > icol[4]=2 > icol[5]=1 > icol[6]=2 > icol[7]=3 > ival[0]=1.000000 > ival[1]=2.000000 > ival[2]=2.000000 > ival[3]=5.000000 > ival[4]=3.000000 > ival[5]=1.000000 > ival[6]=4.000000 > ival[7]=1.000000 > Matrix A: > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 1) (1, 2) > row 1: (0, 2) (2, 5) > row 2: (1, 3) > row 3: (0, 1) (1, 4) (2, 1) > Vector b: > Vector Object: 1 MPI processes > type: seq > 1 > 2 > 3 > 4 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Must be square matrix, rows 4 columns 3! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ss on a arch-linu named olympus.local by d3m956 Fri Aug > 3 15:34:07 2012 > [0]PETSC ERROR: Libraries linked from > /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 228 in src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 206 in src/ksp/pc/impls/factor/ilu/ilu.c > > It is ILU, the default preconditioner, that requires a square matrix. Try > -pc_type none > > Matt > > [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c > KSPGetIterationNumber 0 > KSPGetResidualNorm 0.000000 > KSPConvergedReason 0 > Vector Object: 1 MPI processes > type: seq > 0 > 0 > 0 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using diagonal shift to prevent zero pivot > matrix ordering: natural > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=4, cols=3 > total: nonzeros=8, allocated nonzeros=8 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > I don't understand why the lsqr solver requires a square matrix "Must be > square matrix, rows 4 columns 3!" > > Can anyone look into the problem for me? > > Thanks a lot! > > Shuangshuang > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pinna.2006 at gmail.com Sun Aug 5 17:19:25 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Mon, 6 Aug 2012 00:19:25 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: Message-ID: Thank you for your quick reply,I will try with a reboot. Regarding the compiler my idea was to use MSVC express 2008. I thought it was possible to compile both library with gcc under cygwin. >From your reply I guess this is not the case. Does this mean I should compile Petsc with MSVC express as well, isn't it? Fabio. 2012/8/5 Satish Balay : > On Sun, 5 Aug 2012, Fabio wrote: > >> Dear all, >> I am new to petsc, and I've been struggling with it for the last days. >> >> I want to install it on Windows 7, and first I tried to install the >> Microsoft MPI and then compiling the MPICH. They both seems ok but I >> am not able to compile Petsc with any of them. >> I am mostly interested to MPICH (for portability). >> I used this script to configure >> >> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 >> --download-f2cblaslapack=1 >> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 > > What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? > > [MS MPI is for use with MS compiler only. MPICH binary install is for > MS compilers, and MPICH source install is for cygwin gcc/g++] > >> >> and I've got this error, >> >> UNABLE to EXECUTE BINARIES for config/configure.py >> --------------------------------------------------------------------------------------- >> [Errno 16] Device or resource busy: 'conftest.exe' >> >> This error seems to happen pretty randomly at differente places in the script. >> Any idea to solve this problem? > > Its bad file locks on windows. The only suggestion we have is reboot. > > For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be > that issue. > > If you still have issues - send the complete log [and logfile] to petsc-maint. > > Satish > >> Thank you all >> >> Fabio >> From bsmith at mcs.anl.gov Sun Aug 5 17:31:10 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 5 Aug 2012 17:31:10 -0500 Subject: [petsc-users] Configure error In-Reply-To: References: Message-ID: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> On Aug 5, 2012, at 5:19 PM, Fabio wrote: > Thank you for your quick reply,I will try with a reboot. > > Regarding the compiler my idea was to use MSVC express 2008. I thought > it was possible to compile both library with gcc under cygwin. > From your reply I guess this is not the case. > Does this mean I should compile Petsc with MSVC express as well, isn't it? Yes, you need to compile PETSc with the compiler you would like to use and make sure to use the MPICH binary install since that works with the Microsoft compilers. Barry > > > Fabio. > > > > 2012/8/5 Satish Balay : >> On Sun, 5 Aug 2012, Fabio wrote: >> >>> Dear all, >>> I am new to petsc, and I've been struggling with it for the last days. >>> >>> I want to install it on Windows 7, and first I tried to install the >>> Microsoft MPI and then compiling the MPICH. They both seems ok but I >>> am not able to compile Petsc with any of them. >>> I am mostly interested to MPICH (for portability). >>> I used this script to configure >>> >>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 >>> --download-f2cblaslapack=1 >>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 >> >> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? >> >> [MS MPI is for use with MS compiler only. MPICH binary install is for >> MS compilers, and MPICH source install is for cygwin gcc/g++] >> >>> >>> and I've got this error, >>> >>> UNABLE to EXECUTE BINARIES for config/configure.py >>> --------------------------------------------------------------------------------------- >>> [Errno 16] Device or resource busy: 'conftest.exe' >>> >>> This error seems to happen pretty randomly at differente places in the script. >>> Any idea to solve this problem? >> >> Its bad file locks on windows. The only suggestion we have is reboot. >> >> For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be >> that issue. >> >> If you still have issues - send the complete log [and logfile] to petsc-maint. >> >> Satish >> >>> Thank you all >>> >>> Fabio >>> From abhyshr at mcs.anl.gov Sun Aug 5 20:22:23 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Sun, 5 Aug 2012 20:22:23 -0500 (CDT) Subject: [petsc-users] error when using ksplsqr for overdetermined system In-Reply-To: Message-ID: <2094380407.162004.1344216143104.JavaMail.root@zimbra-mb2.anl.gov> i) The problem you are trying to solve did not converge, look at the number of KSP iterations (10000 is the default max) and the residual norm. ii) You are not solving the problem in parallel but solving the same instance of the problem on each processor. Here's what you have in your code. comm = MPI_COMM_SELF; I suggest storing your matrix in binary format first and then using MatLoad() http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatLoad.html to load your matrix in parallel, and then calling KSP. The communicator should be MPI_COMM_WORLD and not MPI_COMM_SELF. As a start, you can modify the example http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex27.c.html to suit what you are trying to do. Btw, is this a power system state estimation problem that you are trying to solve? Shri ----- Original Message ----- > Hi, Mat, I replaced the inputs with a larger system (the A matrix has > 27090 rows and 14959 columns). With the -pc_type none set up as the > runtime option, the system can be solved correctly, but the new > problem is the solving time is very slow, taking about 75 seconds to > run KSPSolve for the lsqr problem. Is there any faster preconditioning > type we can use here instead of "none"? > > [d3m956 at olympus tutorials]$ mpiexec -n 1 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.195198 > KSPSolve Elapsed Time: 74.788414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 75.155196 > > Another question is how to run lsqr in parallel. I tried to use 4 > processors for example, but it didn't help to speed up. > > [d3m956 at olympus tutorials]$ mpiexec -n 4 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.177819 > Pre KSPSolve Elapsed Time: 0.180807 > Pre KSPSolve Elapsed Time: 0.179783 > Pre KSPSolve Elapsed Time: 0.181975 > KSPSolve Elapsed Time: 73.107033 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.455209 > KSPSolve Elapsed Time: 73.425921 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.771948 > KSPSolve Elapsed Time: 76.629414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSPSolve Elapsed Time: 76.698510 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.174527 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.247951 > > Is there anything I need to do in the code to parallelize the ksp lsqr > solver other than specify the mpiexec -n option in > the command line? > > Thanks, > Shuangshuang > > > On 8/3/12 4:35 PM, "Matthew Knepley" wrote: > > On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang > wrote: > Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. > I have read the following threads regarding the topic: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html > > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html > > > I defined a 4*3 matrix A and a vector b in sparse CSR format: > irow: 1 3 5 6 9 > icol: 1 2 1 3 2 1 2 3 > ival: 1 2 2 5 3 1 4 1 > rhs: 1 2 3 4 > > A = 1 2 0 > 2 0 5 > 0 3 0 > 1 4 1 > b = 1 > 2 > 3 > 4 > > I wrote the following code to solve this system: > > static char help[]="Reading in a matrix\n"; > > #include > #include > #include > #include "petscvec.h" /* This enables us to use vectors. */ > #include "petscmat.h" /* This enables us to use Matrices. It includes > the petscvec header file*/ > #include "petscksp.h" /* Now we can solve linear systems. Solvers used > are KSP. */ > > int main(int argc, char **argv) > { > /* Declaration of Matrix A and some vectors x*/ > Vec b,x; > Mat A; > FILE *fp; > > MPI_Comm comm; > PetscInt n=4,m=3,nz=8,index; > PetscErrorCode ierr; > PetscInt i,j; > comm = MPI_COMM_SELF; > PetscInt irow[n+1],icol[nz]; > PetscScalar ival[nz],rhs[n]; > PetscInt *cnt,*col; > > KSP ksp; > > /* This part is needed for the help flag supplied at run-time*/ > ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); > ierr = > PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); > > /*===================================================================================*/ > > fp=fopen("irow.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n+1; i++) > { > if (fscanf(fp,"%d", &irow[i]) != 1) > { > fprintf(stderr, "Failed to read irow vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); > > fp=fopen("icol.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%d", &icol[i]) != 1) > { > fprintf(stderr, "Failed to read icol vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); > > fp=fopen("ival.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%lf", &ival[i]) != 1) > { > fprintf(stderr, "Failed to read ival vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); > > /*===================================================================================*/ > > // determine number of nonzeros per row in the new matrix > PetscMalloc((n+1)*sizeof(PetscInt),&cnt); > for (i=0;i cnt[i]=irow[i+1]-irow[i]; > } > MatCreateSeqAIJ(PETSC_COMM_SELF,n,m,0,cnt,&A); > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); > // copy over the matrix entries from the inputs > for (i=0;i PetscMalloc(cnt[i]*sizeof(PetscInt),&col); > for (j=0;j col[j]=icol[irow[i]+j-1]-1; > } > for (j=irow[i]-1;j MatSetValues(A,1,&i,cnt[i],col,&ival[irow[i]-1],INSERT_VALUES); > } > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > ierr=PetscPrintf(PETSC_COMM_WORLD,"Matrix A:\n");CHKERRQ(ierr); > MatView(A,PETSC_VIEWER_STDOUT_SELF); > > /*===================================================================================*/ > > /* Use options from the terminal to create a vector that is type > shared or mpi. */ > ierr = VecCreate(PETSC_COMM_WORLD,&b);CHKERRQ(ierr); /* Vector > creation */ > ierr = VecSetSizes(b,PETSC_DECIDE,n);CHKERRQ(ierr); /* Setting the > vector size */ > ierr = VecSetFromOptions(b);CHKERRQ(ierr); /* Setting the vector type > (shared, mpi etc) */ > /* The second argument is a PETSc scalar value.*/ > ierr = VecSet(b,0);CHKERRQ(ierr); > > /* Reading in the RHS vector. */ > /* Reading in the matrix from the file matrix.txt */ > //fp=fopen("b1.mat","r"); > fp=fopen("rhs.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n; i++) > { > if (fscanf(fp,"%lf", &rhs[i]) != 1) > { > fprintf(stderr, "Failed to read rhs vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > > index=0; > /*Putting x into final form */ > for (i=0; i { > /* Oneinsertion per step. Thats what the 1 in second argument stands > for */ > ierr= VecSetValues(b,1,&index,&rhs[i],INSERT_VALUES);CHKERRQ(ierr); > index=index+1; > } /* The third and fourth arguments are addresses. The fifth argument > is IORA */ > > /* Assembling the vector. */ > ierr= VecAssemblyBegin(b);CHKERRQ(ierr); > ierr=VecAssemblyEnd(b);CHKERRQ(ierr); > > /* Viewing the changed vector. */ > ierr=PetscPrintf(PETSC_COMM_WORLD,"Vector b:\n");CHKERRQ(ierr); > ierr=VecView(b,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > /*===================================================================================*/ > > /* solve kspslqr system */ > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,m); > VecSetFromOptions(x); > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPLSQR); > KSPSetTolerances(ksp,1.0e-6,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetOperators(ksp, A, A, SAME_PRECONDITIONER); > KSPSetFromOptions(ksp); > KSPSolve(ksp, b, x); > > PetscInt num_iters; > KSPGetIterationNumber(ksp, &num_iters); CHKERRQ(ierr); > > PetscReal rnorm; > ierr = KSPGetResidualNorm(ksp, &rnorm); CHKERRQ(ierr); > > KSPConvergedReason reason; > ierr = KSPGetConvergedReason(ksp, &reason); CHKERRQ(ierr); > > printf("KSPGetIterationNumber %d\n",num_iters); > printf("KSPGetResidualNorm %f\n",rnorm); > printf("KSPConvergedReason %d\n",reason); > > //VecView(x,PETSC_VIEWER_STDOUT_WORLD); > //PetscViewerASCIIOpen(PETSC_COMM_WORLD, "x.data", &viewer); > VecView(x,PETSC_VIEWER_STDOUT_WORLD); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > /* > Destroy any objects created. > */ > ierr=VecDestroy(&b);CHKERRQ(ierr); > ierr=MatDestroy(&A);CHKERRQ(ierr); > ierr=PetscFinalize();CHKERRQ(ierr); > > return 0; > } > > This code read the inputs data and formed the A and b matrix correctly > as I can view them in the output. However, I got "Invalid argument!" > error as shown below: > > [d3m956 at olympus tutorials]$ ./ss > irow[0]=1 > irow[1]=3 > irow[2]=5 > irow[3]=6 > irow[4]=9 > icol[0]=1 > icol[1]=2 > icol[2]=1 > icol[3]=3 > icol[4]=2 > icol[5]=1 > icol[6]=2 > icol[7]=3 > ival[0]=1.000000 > ival[1]=2.000000 > ival[2]=2.000000 > ival[3]=5.000000 > ival[4]=3.000000 > ival[5]=1.000000 > ival[6]=4.000000 > ival[7]=1.000000 > Matrix A: > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 1) (1, 2) > row 1: (0, 2) (2, 5) > row 2: (1, 3) > row 3: (0, 1) (1, 4) (2, 1) > Vector b: > Vector Object: 1 MPI processes > type: seq > 1 > 2 > 3 > 4 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Must be square matrix, rows 4 columns 3! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 > 14:20:42 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ss on a arch-linu named olympus.local by d3m956 Fri > Aug 3 15:34:07 2012 > [0]PETSC ERROR: Libraries linked from > /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 228 in src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 206 in > src/ksp/pc/impls/factor/ilu/ilu.c > > It is ILU, the default preconditioner, that requires a square matrix. > Try -pc_type none > > Matt > > [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c > KSPGetIterationNumber 0 > KSPGetResidualNorm 0.000000 > KSPConvergedReason 0 > Vector Object: 1 MPI processes > type: seq > 0 > 0 > 0 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using diagonal shift to prevent zero pivot > matrix ordering: natural > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=4, cols=3 > total: nonzeros=8, allocated nonzeros=8 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > I don't understand why the lsqr solver requires a square matrix "Must > be square matrix, rows 4 columns 3!" > > Can anyone look into the problem for me? > > Thanks a lot! > > Shuangshuang From zengshixiangze at 163.com Sun Aug 5 23:44:15 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Mon, 6 Aug 2012 12:44:15 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> Message-ID: <4a7bd0.cbd3.138fa3f686c.Coremail.zengshixiangze@163.com> Do you mean all the computational work are done on the GPU? When I run ex5 with -dm_vec_type veccusp -dm_mat_type mataijcusp, it appears the following error: ~/ex5\>./ex5 -dm_vec_type veccusp -dm_mat_type -log_summary ex5_log [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! [0]PETSC ERROR: Unknown vector type: veccusp! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex5 on a arch-cuda named hohhot by hongwang Mon Aug 6 12:27:19 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecSetType() line 44 in src/vec/vec/interface/vecreg.c [0]PETSC ERROR: DMCreateGlobalVector_DA() line 36 in src/dm/impls/da/dadist.c [0]PETSC ERROR: DMCreateGlobalVector() line 443 in src/dm/interface/dm.c [0]PETSC ERROR: DMDASetUniformCoordinates() line 58 in src/dm/impls/da/gr1.c [0]PETSC ERROR: main() line 113 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 Is there something wrong with the CUSP? My PETSc version is -dev, the cusp version I use is 0.3.1, CUDA version is 4.2. Zeng Xiangze ? 2012-08-06 03:18:58?"Matthew Knepley" ??? On Sun, Aug 5, 2012 at 10:24 AM, Xiangze Zeng wrote: Dear Matt, Thank you for your suggestion. I'm learning to use the GPU effectively step by step. I think it's useful for the novice if there is a manual about using PETSc with CUDA. Each iteration is done, the VEC will be copied to the host to evaluate the stopping condition, is it right? No, if that was true, we would have given up long ago. My guess is that some of your Vecs are not the correct type. Can you look at ex5 suing -dm_vec_type veccusp -dm_mat_type mataijcusp and mail petsc-maint at mcs.anl.gov? Matt Sincerely, Zeng Xiangze ? 2012-08-05 20:27:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng wrote: When I change the PC type to JACOBI, the KSP type to BICG, although the computational speed both in the GPU and CPU are higher than that when I use SOR+BCGS, the computational work in the GPU doesn't seem much more efficient, the speed only 20% higher. Is there any proposal? The attachments are the output of the log_summary. You also have to look at the log_summary: VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1) I said to use GMRES for a reason. Listen to me. BiCG uses the transpose, which right now confuses the results 2) Look at the copies to/from the GPU. You should not be copying the vector 4000 times. Start simple until you understand everything about how the code is running. Use -pc_type none -ksp_type gmres and see if you can understand the results. Then try different KSP and PC. Trying everything at once does not help anyone, and it is not science. Matt Thank you! Zeng Xiangze At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the manual, is it KSPBCGS? ? 2012-08-04 23:04:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. This is breakdown in that algorithm. Try GMRES. Matt Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Aug 5 23:46:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 5 Aug 2012 22:46:57 -0600 Subject: [petsc-users] [petsc-maint #127209] Re: Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> <4a7bd0.cbd3.138fa3f686c.Coremail.zengshixiangze@163.com> Message-ID: On Sun, Aug 5, 2012 at 10:44 PM, Xiangze Zeng wrote: > Do you mean all the computational work are done on the GPU? > > > When I run ex5 with -dm_vec_type veccusp -dm_mat_type mataijcusp, it > appears the following error: > -dm_vec_type cusp -dm_mat_type aijcusp > > > ~/ex5\>./ex5 -dm_vec_type veccusp -dm_mat_type -log_summary ex5_log > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type: > see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > [0]PETSC ERROR: Unknown vector type: veccusp! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 > -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex5 on a arch-cuda named hohhot by hongwang Mon Aug 6 > 12:27:19 2012 > [0]PETSC ERROR: Libraries linked from > /usr/src/petsc/petsc-dev/arch-cuda-double/lib > [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 > [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 > --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 > --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 > --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 > --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 > --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 > --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 > --with-clanguage=c --with-cuda-arch=sm_20 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecSetType() line 44 in src/vec/vec/interface/vecreg.c > [0]PETSC ERROR: DMCreateGlobalVector_DA() line 36 in > src/dm/impls/da/dadist.c > [0]PETSC ERROR: DMCreateGlobalVector() line 443 in src/dm/interface/dm.c > [0]PETSC ERROR: DMDASetUniformCoordinates() line 58 in > src/dm/impls/da/gr1.c > [0]PETSC ERROR: main() line 113 in src/snes/examples/tutorials/ex5.c > application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 > > > Is there something wrong with the CUSP? My PETSc version is -dev, the cusp > version I use is 0.3.1, CUDA version is 4.2. > > > Zeng Xiangze > ? 2012-08-06 03:18:58?"Matthew Knepley" ??? > On Sun, Aug 5, 2012 at 10:24 AM, Xiangze Zeng > wrote: > > Dear Matt, > > > Thank you for your suggestion. I'm learning to use the GPU effectively > step by step. I think it's useful for the novice if there is a manual about > using PETSc with CUDA. > Each iteration is done, the VEC will be copied to the host to evaluate the > stopping condition, is it right? > > > No, if that was true, we would have given up long ago. My guess is that > some of your Vecs are not the correct type. > Can you look at ex5 suing -dm_vec_type veccusp -dm_mat_type mataijcusp and > mail petsc-maint at mcs.anl.gov? > > > Matt > > Sincerely, > Zeng Xiangze > > > > ? 2012-08-05 20:27:55?"Matthew Knepley" ??? > On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng > wrote: > > When I change the PC type to JACOBI, the KSP type to BICG, although the > computational speed both in the GPU and CPU are higher than that when I use > SOR+BCGS, the computational work in the GPU doesn't seem much more > efficient, the speed only 20% higher. Is there any proposal? The > attachments are the output of the log_summary. > > > You also have to look at the log_summary: > > > VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 > MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > 1) I said to use GMRES for a reason. Listen to me. BiCG uses the > transpose, which right now confuses the results > > > 2) Look at the copies to/from the GPU. You should not be copying the > vector 4000 times. Start simple until you understand > everything about how the code is running. Use -pc_type none -ksp_type > gmres and see if you can understand the results. > Then try different KSP and PC. Trying everything at once does not help > anyone, and it is not science. > > > Matt > > Thank you! > > > Zeng Xiangze > > At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: > > JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. > When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In > http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov > methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the > manual, is it KSPBCGS? > > ? 2012-08-04 23:04:55?"Matthew Knepley" ??? > On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng > wrote: > > Another error happens when I change the PC type. When I change it to > PCJACOBI, it appears the following error message: > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: Divide by zero! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 > -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by > hongwang Sat Aug 4 22:23:58 2012 > [0]PETSC ERROR: Libraries linked from > /usr/src/petsc/petsc-dev/arch-cuda-double/lib > [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 > [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 > --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 > --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 > --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 > --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 > --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 > --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 > --with-clanguage=c --with-cuda-arch=sm_20 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c > > > And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of > memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the > result is not better than that when I don't change the type. > > > This is breakdown in that algorithm. Try GMRES. > > > Matt > > Does it have something to do with the KSP type? Should I look for a suited > KSP type to match the PC type which can work on the GPU? > > ? 2012-08-04 21:44:02?"Matthew Knepley" ??? > On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng > wrote: > > After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 > minutes, a little better than before. The attachment are the output of > -log_summary. > > > 1) Notice how the PCApply takes most of the time, so MatMult is not very > important > > > 2) In g_log_3, notice that every time your PC is called, the vector is > pulled from the GPU to the CPU. > This means we do not support that PC on the GPU > > > There is a restriction on PCs since not many are coded for the GPU. Only > PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP > work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. > > > Matt > > At 2012-08-04 14:40:33,"Azamat Mametjanov" > wrote: > What happens if you try to re-run with "--with-debugging=no"? > > > On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng > wrote: > > Dear Matt, > > > My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. > The size of the system is 2522469 x 2522469, and the number non-0 elements > is 71773925, about 0.000012 of the total. > The output of -log_summary is in the attachment. The G_log_summary is the > output when using GPU, C_log_summary when using CPU. > > > Zeng Xiangze > > > ? 2012-08-03 22:28:07?"Matthew Knepley" ??? > > On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng > wrote: > > Dear all, > > > When I use the CPU solve the equations, it takes 78 minutes, when I change > to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper > say when using PETCs with GPU to solve the large sparse matrix equations, > it can be several times faster? What's the matter? > > > For all performance questions, we at least need the output of > -log_summary. However, we would also need to know > > > - The size and sparsity of your system > > > - The CPU and GPU you used (saying anything without knowing this is > impossible) > > > Matt > > Thank you! > > > Sincerely, > Zeng Xiangze > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > > > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Mailbox 379, School of Physics > Shandong University > 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Sun Aug 5 23:53:30 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Sun, 5 Aug 2012 21:53:30 -0700 Subject: [petsc-users] error when using ksplsqr for overdetermined system In-Reply-To: <2094380407.162004.1344216143104.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: Hi, Shri, thanks for your message. The information is very useful to me. I'll look into the problem again and try your recommendations. And yes, this is a power system state estimation problem we are trying to solve. Thanks, Shuangshuang On 8/5/12 6:22 PM, "Shri" wrote: i) The problem you are trying to solve did not converge, look at the number of KSP iterations (10000 is the default max) and the residual norm. ii) You are not solving the problem in parallel but solving the same instance of the problem on each processor. Here's what you have in your code. comm = MPI_COMM_SELF; I suggest storing your matrix in binary format first and then using MatLoad() http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatLoad.html to load your matrix in parallel, and then calling KSP. The communicator should be MPI_COMM_WORLD and not MPI_COMM_SELF. As a start, you can modify the example http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex27.c.html to suit what you are trying to do. Btw, is this a power system state estimation problem that you are trying to solve? Shri ----- Original Message ----- > Hi, Mat, I replaced the inputs with a larger system (the A matrix has > 27090 rows and 14959 columns). With the -pc_type none set up as the > runtime option, the system can be solved correctly, but the new > problem is the solving time is very slow, taking about 75 seconds to > run KSPSolve for the lsqr problem. Is there any faster preconditioning > type we can use here instead of "none"? > > [d3m956 at olympus tutorials]$ mpiexec -n 1 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.195198 > KSPSolve Elapsed Time: 74.788414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 75.155196 > > Another question is how to run lsqr in parallel. I tried to use 4 > processors for example, but it didn't help to speed up. > > [d3m956 at olympus tutorials]$ mpiexec -n 4 ss -pc_type none > Pre KSPSolve Elapsed Time: 0.177819 > Pre KSPSolve Elapsed Time: 0.180807 > Pre KSPSolve Elapsed Time: 0.179783 > Pre KSPSolve Elapsed Time: 0.181975 > KSPSolve Elapsed Time: 73.107033 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.455209 > KSPSolve Elapsed Time: 73.425921 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 73.771948 > KSPSolve Elapsed Time: 76.629414 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSPSolve Elapsed Time: 76.698510 > KSPGetIterationNumber 10000 > KSPGetResidualNorm 17921.356837 > KSPConvergedReason -3 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.174527 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: none > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=27090, cols=14959 > total: nonzeros=70903, allocated nonzeros=70903 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Overall Elapsed Time: 77.247951 > > Is there anything I need to do in the code to parallelize the ksp lsqr > solver other than specify the mpiexec -n option in > the command line? > > Thanks, > Shuangshuang > > > On 8/3/12 4:35 PM, "Matthew Knepley" wrote: > > On Fri, Aug 3, 2012 at 5:38 PM, Jin, Shuangshuang > wrote: > Hi, all, I'm trying to use ksplsqr to solve an overdetermined system. > I have read the following threads regarding the topic: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007763.html > > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-August/006784.html > > > I defined a 4*3 matrix A and a vector b in sparse CSR format: > irow: 1 3 5 6 9 > icol: 1 2 1 3 2 1 2 3 > ival: 1 2 2 5 3 1 4 1 > rhs: 1 2 3 4 > > A = 1 2 0 > 2 0 5 > 0 3 0 > 1 4 1 > b = 1 > 2 > 3 > 4 > > I wrote the following code to solve this system: > > static char help[]="Reading in a matrix\n"; > > #include > #include > #include > #include "petscvec.h" /* This enables us to use vectors. */ > #include "petscmat.h" /* This enables us to use Matrices. It includes > the petscvec header file*/ > #include "petscksp.h" /* Now we can solve linear systems. Solvers used > are KSP. */ > > int main(int argc, char **argv) > { > /* Declaration of Matrix A and some vectors x*/ > Vec b,x; > Mat A; > FILE *fp; > > MPI_Comm comm; > PetscInt n=4,m=3,nz=8,index; > PetscErrorCode ierr; > PetscInt i,j; > comm = MPI_COMM_SELF; > PetscInt irow[n+1],icol[nz]; > PetscScalar ival[nz],rhs[n]; > PetscInt *cnt,*col; > > KSP ksp; > > /* This part is needed for the help flag supplied at run-time*/ > ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr); > ierr = > PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);CHKERRQ(ierr); > > /*===================================================================================*/ > > fp=fopen("irow.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n+1; i++) > { > if (fscanf(fp,"%d", &irow[i]) != 1) > { > fprintf(stderr, "Failed to read irow vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < n+1; i++) printf("irow[%d]=%d\n",i,irow[i]); > > fp=fopen("icol.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%d", &icol[i]) != 1) > { > fprintf(stderr, "Failed to read icol vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("icol[%d]=%d\n",i,icol[i]); > > fp=fopen("ival.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < nz; i++) > { > if (fscanf(fp,"%lf", &ival[i]) != 1) > { > fprintf(stderr, "Failed to read ival vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > for (i = 0; i < nz; i++) printf("ival[%d]=%lf\n",i,ival[i]); > > /*===================================================================================*/ > > // determine number of nonzeros per row in the new matrix > PetscMalloc((n+1)*sizeof(PetscInt),&cnt); > for (i=0;i cnt[i]=irow[i+1]-irow[i]; > } > MatCreateSeqAIJ(PETSC_COMM_SELF,n,m,0,cnt,&A); > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); > // copy over the matrix entries from the inputs > for (i=0;i PetscMalloc(cnt[i]*sizeof(PetscInt),&col); > for (j=0;j col[j]=icol[irow[i]+j-1]-1; > } > for (j=irow[i]-1;j MatSetValues(A,1,&i,cnt[i],col,&ival[irow[i]-1],INSERT_VALUES); > } > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > ierr=PetscPrintf(PETSC_COMM_WORLD,"Matrix A:\n");CHKERRQ(ierr); > MatView(A,PETSC_VIEWER_STDOUT_SELF); > > /*===================================================================================*/ > > /* Use options from the terminal to create a vector that is type > shared or mpi. */ > ierr = VecCreate(PETSC_COMM_WORLD,&b);CHKERRQ(ierr); /* Vector > creation */ > ierr = VecSetSizes(b,PETSC_DECIDE,n);CHKERRQ(ierr); /* Setting the > vector size */ > ierr = VecSetFromOptions(b);CHKERRQ(ierr); /* Setting the vector type > (shared, mpi etc) */ > /* The second argument is a PETSc scalar value.*/ > ierr = VecSet(b,0);CHKERRQ(ierr); > > /* Reading in the RHS vector. */ > /* Reading in the matrix from the file matrix.txt */ > //fp=fopen("b1.mat","r"); > fp=fopen("rhs.txt","r"); > if (fp==NULL) > { > fprintf(stderr, "Cannot open file\n"); > exit(1); > } > > for (i = 0; i < n; i++) > { > if (fscanf(fp,"%lf", &rhs[i]) != 1) > { > fprintf(stderr, "Failed to read rhs vector[%d]\n", i); > exit(1); > } > } > fclose(fp); > > index=0; > /*Putting x into final form */ > for (i=0; i { > /* Oneinsertion per step. Thats what the 1 in second argument stands > for */ > ierr= VecSetValues(b,1,&index,&rhs[i],INSERT_VALUES);CHKERRQ(ierr); > index=index+1; > } /* The third and fourth arguments are addresses. The fifth argument > is IORA */ > > /* Assembling the vector. */ > ierr= VecAssemblyBegin(b);CHKERRQ(ierr); > ierr=VecAssemblyEnd(b);CHKERRQ(ierr); > > /* Viewing the changed vector. */ > ierr=PetscPrintf(PETSC_COMM_WORLD,"Vector b:\n");CHKERRQ(ierr); > ierr=VecView(b,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > /*===================================================================================*/ > > /* solve kspslqr system */ > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,m); > VecSetFromOptions(x); > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPLSQR); > KSPSetTolerances(ksp,1.0e-6,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetOperators(ksp, A, A, SAME_PRECONDITIONER); > KSPSetFromOptions(ksp); > KSPSolve(ksp, b, x); > > PetscInt num_iters; > KSPGetIterationNumber(ksp, &num_iters); CHKERRQ(ierr); > > PetscReal rnorm; > ierr = KSPGetResidualNorm(ksp, &rnorm); CHKERRQ(ierr); > > KSPConvergedReason reason; > ierr = KSPGetConvergedReason(ksp, &reason); CHKERRQ(ierr); > > printf("KSPGetIterationNumber %d\n",num_iters); > printf("KSPGetResidualNorm %f\n",rnorm); > printf("KSPConvergedReason %d\n",reason); > > //VecView(x,PETSC_VIEWER_STDOUT_WORLD); > //PetscViewerASCIIOpen(PETSC_COMM_WORLD, "x.data", &viewer); > VecView(x,PETSC_VIEWER_STDOUT_WORLD); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > /* > Destroy any objects created. > */ > ierr=VecDestroy(&b);CHKERRQ(ierr); > ierr=MatDestroy(&A);CHKERRQ(ierr); > ierr=PetscFinalize();CHKERRQ(ierr); > > return 0; > } > > This code read the inputs data and formed the A and b matrix correctly > as I can view them in the output. However, I got "Invalid argument!" > error as shown below: > > [d3m956 at olympus tutorials]$ ./ss > irow[0]=1 > irow[1]=3 > irow[2]=5 > irow[3]=6 > irow[4]=9 > icol[0]=1 > icol[1]=2 > icol[2]=1 > icol[3]=3 > icol[4]=2 > icol[5]=1 > icol[6]=2 > icol[7]=3 > ival[0]=1.000000 > ival[1]=2.000000 > ival[2]=2.000000 > ival[3]=5.000000 > ival[4]=3.000000 > ival[5]=1.000000 > ival[6]=4.000000 > ival[7]=1.000000 > Matrix A: > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 1) (1, 2) > row 1: (0, 2) (2, 5) > row 2: (1, 3) > row 3: (0, 1) (1, 4) (2, 1) > Vector b: > Vector Object: 1 MPI processes > type: seq > 1 > 2 > 3 > 4 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Must be square matrix, rows 4 columns 3! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 > 14:20:42 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ss on a arch-linu named olympus.local by d3m956 Fri > Aug 3 15:34:07 2012 > [0]PETSC ERROR: Libraries linked from > /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 228 in src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 206 in > src/ksp/pc/impls/factor/ilu/ilu.c > > It is ILU, the default preconditioner, that requires a square matrix. > Try -pc_type none > > Matt > > [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c > KSPGetIterationNumber 0 > KSPGetResidualNorm 0.000000 > KSPConvergedReason 0 > Vector Object: 1 MPI processes > type: seq > 0 > 0 > 0 > KSP Object: 1 MPI processes > type: lsqr > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using diagonal shift to prevent zero pivot > matrix ordering: natural > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=4, cols=3 > total: nonzeros=8, allocated nonzeros=8 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > I don't understand why the lsqr solver requires a square matrix "Must > be square matrix, rows 4 columns 3!" > > Can anyone look into the problem for me? > > Thanks a lot! > > Shuangshuang From zengshixiangze at 163.com Mon Aug 6 00:06:41 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Mon, 6 Aug 2012 13:06:41 +0800 (CST) Subject: [petsc-users] Why does GPU solve the large sparse matrix equations only a little faster than CPU? In-Reply-To: References: <5cecdde1.11305.138ecd9ce3e.Coremail.zengshixiangze@163.com> <1cdd6f18.15c0b.138ef93f086.Coremail.zengshixiangze@163.com> <1cf30924.1b540.138f1491f58.Coremail.zengshixiangze@163.com> <3b464f98.1cf08.138f216c7e3.Coremail.zengshixiangze@163.com> <7d56cef6.1d4d4.138f25e6ef6.Coremail.zengshixiangze@163.com> <69b910cd.3a9f.138f505b657.Coremail.zengshixiangze@163.com> Message-ID: <1765f262.cf9c.138fa53f0c8.Coremail.zengshixiangze@163.com> The attachment is the output of -log_summary running ex5 with -dm_vec_type cusp -dm_mat_type aijcusp. ? 2012-08-06 03:18:58?"Matthew Knepley" ??? On Sun, Aug 5, 2012 at 10:24 AM, Xiangze Zeng wrote: Dear Matt, Thank you for your suggestion. I'm learning to use the GPU effectively step by step. I think it's useful for the novice if there is a manual about using PETSc with CUDA. Each iteration is done, the VEC will be copied to the host to evaluate the stopping condition, is it right? No, if that was true, we would have given up long ago. My guess is that some of your Vecs are not the correct type. Can you look at ex5 suing -dm_vec_type veccusp -dm_mat_type mataijcusp and mail petsc-maint at mcs.anl.gov? Matt Sincerely, Zeng Xiangze ? 2012-08-05 20:27:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 11:23 PM, Xiangze Zeng wrote: When I change the PC type to JACOBI, the KSP type to BICG, although the computational speed both in the GPU and CPU are higher than that when I use SOR+BCGS, the computational work in the GPU doesn't seem much more efficient, the speed only 20% higher. Is there any proposal? The attachments are the output of the log_summary. You also have to look at the log_summary: VecCUSPCopyTo 3967 1.0 1.3152e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecCUSPCopyFrom 3969 1.0 5.5139e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 MatCUSPCopyTo 1 1.0 4.5194e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 1) I said to use GMRES for a reason. Listen to me. BiCG uses the transpose, which right now confuses the results 2) Look at the copies to/from the GPU. You should not be copying the vector 4000 times. Start simple until you understand everything about how the code is running. Use -pc_type none -ksp_type gmres and see if you can understand the results. Then try different KSP and PC. Trying everything at once does not help anyone, and it is not science. Matt Thank you! Zeng Xiangze At 2012-08-05 00:01:11,"Xiangze Zeng" wrote: JACOBI+GMRES takes 124s to solve one system on the GPU, 172s on the CPU. When I use JACOBI+BICG, it takes 123s on the GPU, 162s on the CPU. In http://www.mcs.anl.gov/petsc/features/gpus.html, I see "All of the Krylov methods except KSPIBCGS run on the GPU. " I don't find KSPIBCGS in the manual, is it KSPBCGS? ? 2012-08-04 23:04:55?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 9:42 AM, Xiangze Zeng wrote: Another error happens when I change the PC type. When I change it to PCJACOBI, it appears the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: d01946145980533f72b6500bd243b1dd3666686c HG Date: Mon Jul 30 17:03:27 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../femsolcu/./femsolcu on a arch-cuda named hohhot by hongwang Sat Aug 4 22:23:58 2012 [0]PETSC ERROR: Libraries linked from /usr/src/petsc/petsc-dev/arch-cuda-double/lib [0]PETSC ERROR: Configure run at Sat Aug 4 15:10:44 2012 [0]PETSC ERROR: Configure options --doCleanup=1 --with-gnu-compilers=1 --with-vendor-compilers=0 --CFLAGS=-march=x86-64 --CXXFLAGS=-march=x86-64 --with-dynamic-loading --with-python=1 --with-debugging=0 --with-log=1 --download-mpich=1 --with-hypre=0 --with-64-bit-indices=yes --with-x11=1 --with-x11-include=/usr/include/X11 --download-f-blas-lapack=1 --with-cuda=1 --with-cusp=1 --with-thrust=1 --download-txpetscgpu=1 --with-precision=double --with-cudac="nvcc -m64" --download-txpetscgpu=1 --with-clanguage=c --with-cuda-arch=sm_20 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 105 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: sol_comp() line 39 in "unknowndirectory/"solve.c And when I change it to PCSACUSP, PCSACUSPPOLY, it both prompts out of memory(I guess it's the GPU's memory). When I change it to PCAINVCUSP, the result is not better than that when I don't change the type. This is breakdown in that algorithm. Try GMRES. Matt Does it have something to do with the KSP type? Should I look for a suited KSP type to match the PC type which can work on the GPU? ? 2012-08-04 21:44:02?"Matthew Knepley" ??? On Sat, Aug 4, 2012 at 5:58 AM, Xiangze Zeng wrote: After I rerun with "deugging=no", the CPU takes 30 minutes, GPU 22 minutes, a little better than before. The attachment are the output of -log_summary. 1) Notice how the PCApply takes most of the time, so MatMult is not very important 2) In g_log_3, notice that every time your PC is called, the vector is pulled from the GPU to the CPU. This means we do not support that PC on the GPU There is a restriction on PCs since not many are coded for the GPU. Only PCJACOBI, PCSACUSP, PCSACUSPPOLY, and PCAINVCUSP work there, see http://www.mcs.anl.gov/petsc/features/gpus.html. Matt At 2012-08-04 14:40:33,"Azamat Mametjanov" wrote: What happens if you try to re-run with "--with-debugging=no"? On Fri, Aug 3, 2012 at 10:00 PM, Xiangze Zeng wrote: Dear Matt, My CPU is Intel Xeon E5-2609, GPU is Nvidia GF100 [Quadro 4000]. The size of the system is 2522469 x 2522469, and the number non-0 elements is 71773925, about 0.000012 of the total. The output of -log_summary is in the attachment. The G_log_summary is the output when using GPU, C_log_summary when using CPU. Zeng Xiangze ? 2012-08-03 22:28:07?"Matthew Knepley" ??? On Fri, Aug 3, 2012 at 9:18 AM, Xiangze Zeng wrote: Dear all, When I use the CPU solve the equations, it takes 78 minutes, when I change to use GPU, it uses 64 minutes, only 15 minutes faster. I see some paper say when using PETCs with GPU to solve the large sparse matrix equations, it can be several times faster? What's the matter? For all performance questions, we at least need the output of -log_summary. However, we would also need to know - The size and sparsity of your system - The CPU and GPU you used (saying anything without knowing this is impossible) Matt Thank you! Sincerely, Zeng Xiangze -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- Mailbox 379, School of Physics Shandong University 27 South Shanda Road, Jinan, Shandong, P.R.China, 250100 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex5_log Type: application/octet-stream Size: 10964 bytes Desc: not available URL: From adam.stanier-2 at postgrad.manchester.ac.uk Mon Aug 6 09:18:10 2012 From: adam.stanier-2 at postgrad.manchester.ac.uk (Adam Stanier) Date: Mon, 6 Aug 2012 14:18:10 +0000 Subject: [petsc-users] FW: Configuring PETSc with intel composer xe 2011, problem with libirc.a In-Reply-To: <7A7F2577D3A89844AB5C35FDD60F77EA0626B8@MBXP07.ds.man.ac.uk> References: <7A7F2577D3A89844AB5C35FDD60F77EA0626B8@MBXP07.ds.man.ac.uk> Message-ID: <7A7F2577D3A89844AB5C35FDD60F77EA0626D7@MBXP07.ds.man.ac.uk> Hello, I'm trying to configure PETSc 3.2-p7 with intel composer xe 2011 mpi libraries. This is the configure line I have been using: ./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blas-lapack-dir=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/ --with-blacs-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a --with-blacs-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include --with-scalapack-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_scalapack_lp64.a --with-scalapack-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include --download-parmetis --download-mumps --download-superlu_dist --download-hypre --with-mpi-dir=/usr/local/intel/impi/4.0.3.008/intel64/ --with-debugging=0 This is fine for making PETSc, but when I try to run a code based on the PETSc libraries I get the following error: ./sel: symbol lookup error: ./sel: undefined symbol: __intel_sse2_strchr I don't get this error when I set --with-debugging=1 with everything else the same. Apparently this symbol should be within the libirc.a library. I notice that PETSc is looking in a folder for gcc for this library: -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 but I want to use the version within the intel lib directory: /usr/local/intel/composer_xe_2011_sp1.6.233/compiler/lib/intel64 Is there a way to do this? I couldn't find anything by searching. Many Thanks, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Mon Aug 6 09:37:43 2012 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Mon, 06 Aug 2012 07:37:43 -0700 Subject: [petsc-users] FW: Configuring PETSc with intel composer xe 2011, problem with libirc.a In-Reply-To: <7A7F2577D3A89844AB5C35FDD60F77EA0626D7@MBXP07.ds.man.ac.uk> References: <7A7F2577D3A89844AB5C35FDD60F77EA0626B8@MBXP07.ds.man.ac.uk> <7A7F2577D3A89844AB5C35FDD60F77EA0626D7@MBXP07.ds.man.ac.uk> Message-ID: <501FD6B7.4010709@berkeley.edu> Did you remember to 'source iccvars.sh' first? On 8/6/12 7:18 AM, Adam Stanier wrote: > > Hello, > > I'm trying to configure PETSc 3.2-p7 with intel composer xe 2011 mpi > libraries. This is the configure line I have been using: > > ./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-blas-lapack-dir=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/ > --with-blacs-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > --with-blacs-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include > --with-scalapack-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_scalapack_lp64.a > --with-scalapack-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include > --download-parmetis --download-mumps --download-superlu_dist > --download-hypre > --with-mpi-dir=/usr/local/intel/impi/4.0.3.008/intel64/ --with-debugging=0 > > This is fine for making PETSc, but when I try to run a code based on > the PETSc libraries I get the following error: > ./sel: symbol lookup error: ./sel: undefined symbol: __intel_sse2_strchr > > I don't get this error when I set --with-debugging=1 with everything > else the same. > > Apparently this symbol should be within the libirc.a library. I notice > that PETSc is looking in a folder for gcc for this library: > -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 > > but I want to use the version within the intel lib directory: > /usr/local/intel/composer_xe_2011_sp1.6.233/compiler/lib/intel64 > > Is there a way to do this? I couldn't find anything by searching. > > Many Thanks, > Adam > > -- ----------------------------------------------- Sanjay Govindjee, PhD, PE Professor of Civil Engineering Vice Chair for Academic Affairs 779 Davis Hall Structural Engineering, Mechanics and Materials Department of Civil Engineering University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://www.ce.berkeley.edu/~sanjay ----------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Mon Aug 6 09:40:31 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Mon, 6 Aug 2012 16:40:31 +0200 Subject: [petsc-users] FW: Configuring PETSc with intel composer xe 2011, problem with libirc.a In-Reply-To: <501FD6B7.4010709@berkeley.edu> References: <7A7F2577D3A89844AB5C35FDD60F77EA0626B8@MBXP07.ds.man.ac.uk> <7A7F2577D3A89844AB5C35FDD60F77EA0626D7@MBXP07.ds.man.ac.uk> <501FD6B7.4010709@berkeley.edu> Message-ID: Here is the answer: http://comments.gmane.org/gmane.comp.clustering.open-mpi.user/12804 On Mon, Aug 6, 2012 at 4:37 PM, Sanjay Govindjee wrote: > Did you remember to 'source iccvars.sh' first? > > > On 8/6/12 7:18 AM, Adam Stanier wrote: > > > Hello, > > I'm trying to configure PETSc 3.2-p7 with intel composer xe 2011 mpi > libraries. This is the configure line I have been using: > > ./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-blas-lapack-dir=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/ > --with-blacs-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > --with-blacs-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include > --with-scalapack-lib=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_scalapack_lp64.a > --with-scalapack-include=/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/include > --download-parmetis --download-mumps --download-superlu_dist > --download-hypre --with-mpi-dir=/usr/local/intel/impi/4.0.3.008/intel64/--with-debugging=0 > > This is fine for making PETSc, but when I try to run a code based on the > PETSc libraries I get the following error: > ./sel: symbol lookup error: ./sel: undefined symbol: __intel_sse2_strchr > > I don't get this error when I set --with-debugging=1 with everything else > the same. > > Apparently this symbol should be within the libirc.a library. I notice > that PETSc is looking in a folder for gcc for this library: > -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 > > but I want to use the version within the intel lib directory: > /usr/local/intel/composer_xe_2011_sp1.6.233/compiler/lib/intel64 > > Is there a way to do this? I couldn't find anything by searching. > > Many Thanks, > Adam > > > > -- > ----------------------------------------------- > Sanjay Govindjee, PhD, PE > Professor of Civil Engineering > Vice Chair for Academic Affairs > > 779 Davis Hall > Structural Engineering, Mechanics and Materials > Department of Civil Engineering > University of California > Berkeley, CA 94720-1710 > > Voice: +1 510 642 6060 > FAX: +1 510 643 5264s_g at berkeley.eduhttp://www.ce.berkeley.edu/~sanjay > ----------------------------------------------- > > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Mon Aug 6 09:36:13 2012 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 6 Aug 2012 22:36:13 +0800 (CST) Subject: [petsc-users] AMS parallel usage Message-ID: <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> Hi, I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. I'd like to know some thing about data synchronization. Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? Gong Ding From andrew at spott.us Mon Aug 6 15:01:09 2012 From: andrew at spott.us (Andrew Spott) Date: Mon, 6 Aug 2012 14:01:09 -0600 Subject: [petsc-users] Finding the inner product of a vector with respect to a matrix. Message-ID: <3E15E9F5-5AD6-4857-88CD-943F8A0D144F@gmail.com> Is there an official way to find v^T A v? Or is it usually just done in two steps? Thanks! -Andrew From knepley at gmail.com Mon Aug 6 16:24:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Aug 2012 16:24:12 -0500 Subject: [petsc-users] Finding the inner product of a vector with respect to a matrix. In-Reply-To: <3E15E9F5-5AD6-4857-88CD-943F8A0D144F@gmail.com> References: <3E15E9F5-5AD6-4857-88CD-943F8A0D144F@gmail.com> Message-ID: On Mon, Aug 6, 2012 at 3:01 PM, Andrew Spott wrote: > Is there an official way to find v^T A v? Or is it usually just done in > two steps? > We do it in two steps. Matt > Thanks! > > -Andrew -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 6 16:41:35 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 6 Aug 2012 16:41:35 -0500 Subject: [petsc-users] AMS parallel usage In-Reply-To: <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> References: <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> Message-ID: On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > Hi, > I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. You need to build AMS with the same MPI that you will use in the parallel program. > > I'd like to know some thing about data synchronization. > Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? There are different types of variables you can publish. AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? Nope, you do not need to do this. Barry > > Gong Ding > > > > > > > From gdiso at ustc.edu Mon Aug 6 22:17:46 2012 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 7 Aug 2012 11:17:46 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: References: <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> Message-ID: <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> Thanks, Barry. It is really a wonderful tool. PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. Hope the AMS manual have some more detailed description to parallel usage and data types. > > > On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > > > > > Hi, > > > I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. > > > > You need to build AMS with the same MPI that you will use in the parallel program. > > > > > > > > I'd like to know some thing about data synchronization. > > > Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? > > > > There are different types of variables you can publish. > > > > AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. > > > > AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . > > > > If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > > > > > > > If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? > > > > AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > > > > > > > If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? > > > > Nope, you do not need to do this. > > > > Barry > > > > > > > > Gong Ding > > > > > > > > > > > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Mon Aug 6 22:28:18 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 6 Aug 2012 22:28:18 -0500 Subject: [petsc-users] AMS parallel usage In-Reply-To: <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> References: <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> Message-ID: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: > Thanks, Barry. It is really a wonderful tool. Thanks > PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. > > Hope the AMS manual have some more detailed description to parallel usage and data types. Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with hg clone http://petsc.cs.iit.edu/petsc/ams-dev Barry > > >> >> >> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: >> >> >> >>> Hi, >> >>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. >> >> >> >> You need to build AMS with the same MPI that you will use in the parallel program. >> >> >> >>> >> >>> I'd like to know some thing about data synchronization. >> >>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? >> >> >> >> There are different types of variables you can publish. >> >> >> >> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. >> >> >> >> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . >> >> >> >> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. >> >> >> >>> >> >>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? >> >> >> >> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. >> >> >> >>> >> >>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? >> >> >> >> Nope, you do not need to do this. >> >> >> >> Barry >> >> >> >>> >> >>> Gong Ding >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >> >> >> From gdiso at ustc.edu Tue Aug 7 01:57:14 2012 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 7 Aug 2012 14:57:14 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> Message-ID: <12080909.277551344322634914.JavaMail.coremail@mail.ustc.edu> OK, I will commit my changes when it is stable. > > > On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: > > > > > Thanks, Barry. It is really a wonderful tool. > > > > Thanks > > > > > PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. > > > > > > Hope the AMS manual have some more detailed description to parallel usage and data types. > > > > Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with > > > > hg clone http://petsc.cs.iit.edu/petsc/ams-dev > > > > > > Barry > > > > > > > > > > >> > > >> > > >> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > > >> > > >> > > >> > > >>> Hi, > > >> > > >>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. > > >> > > >> > > >> > > >> You need to build AMS with the same MPI that you will use in the parallel program. > > >> > > >> > > >> > > >>> > > >> > > >>> I'd like to know some thing about data synchronization. > > >> > > >>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? > > >> > > >> > > >> > > >> There are different types of variables you can publish. > > >> > > >> > > >> > > >> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. > > >> > > >> > > >> > > >> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . > > >> > > >> > > >> > > >> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > >> > > >> > > >> > > >>> > > >> > > >>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? > > >> > > >> > > >> > > >> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > >> > > >> > > >> > > >>> > > >> > > >>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? > > >> > > >> > > >> > > >> Nope, you do not need to do this. > > >> > > >> > > >> > > >> Barry > > >> > > >> > > >> > > >>> > > >> > > >>> Gong Ding > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >> > > >> > > >> > > > > From gdiso at ustc.edu Tue Aug 7 05:25:15 2012 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 7 Aug 2012 18:25:15 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> Message-ID: <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> Some more bugs about AMS_Comm_publish for parallel situation. even the AMS mpi example can not work. 1) parameter mismatch asmpub.c line 88 mysetport = *(int *) va_arg(args, int *); It seems author wants to add another argument as int * to AMS_Comm_publish function for parallel situation. which not appears in the manual. I just comment line 88 and line 130-131, if (mysetport != -1) { AMS_port = mysetport; } 2) extra check of port number asmpub.c line 149 /* Create publisher arguments */ err = AMSP_New_Pub_Arguments(AMS_port, &pub_arg); CheckErr(err); AMSP_New_Pub_Arguments function will check the value of AMS_port. A port number of -1 or 0 will return a error. 3) AMSP_start_publisher_func seems require a port number -1 to force socket bind function to allocate a new port, which conflict with AMSP_New_Pub_Arguments check. nettcppub.c line 66 if (*port == -1) *port = 0; at line 94, a bind function is called. I have to set pub_arg->port = -1 as follows. if(ctype == MPI_TYPE) { pub_arg->care_about_port = 0; pub_arg->port = -1; } else pub_arg->care_about_port = 1; Barry, will you please check it? > > > On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: > > > > > Thanks, Barry. It is really a wonderful tool. > > > > Thanks > > > > > PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. > > > > > > Hope the AMS manual have some more detailed description to parallel usage and data types. > > > > Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with > > > > hg clone http://petsc.cs.iit.edu/petsc/ams-dev > > > > > > Barry > > > > > > > > > > >> > > >> > > >> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > > >> > > >> > > >> > > >>> Hi, > > >> > > >>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. > > >> > > >> > > >> > > >> You need to build AMS with the same MPI that you will use in the parallel program. > > >> > > >> > > >> > > >>> > > >> > > >>> I'd like to know some thing about data synchronization. > > >> > > >>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? > > >> > > >> > > >> > > >> There are different types of variables you can publish. > > >> > > >> > > >> > > >> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. > > >> > > >> > > >> > > >> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . > > >> > > >> > > >> > > >> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > >> > > >> > > >> > > >>> > > >> > > >>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? > > >> > > >> > > >> > > >> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > >> > > >> > > >> > > >>> > > >> > > >>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? > > >> > > >> > > >> > > >> Nope, you do not need to do this. > > >> > > >> > > >> > > >> Barry > > >> > > >> > > >> > > >>> > > >> > > >>> Gong Ding > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >>> > > >> > > >> > > >> > > >> > > > > From w_ang_temp at 163.com Tue Aug 7 08:56:34 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Tue, 7 Aug 2012 21:56:34 +0800 (CST) Subject: [petsc-users] Functions for judging the properties of the matrix In-Reply-To: References: <5daed011.ed92.138764ea964.Coremail.w_ang_temp@163.com> Message-ID: <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> Hello, Jed I am a little puzzled about the "How can I determine the condition number of a matrix?" in the FAQ. The details are as follows: "...For larger matrices, you can run with -pc_type none -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 to get approximations to the condition number of the operator. This will generally be accurate for the largest singular values, but may overestimate the smallest singular value unless the method has converged. Make sure to avoid restarts...". Since the command "-pc_type none -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000" using restart parameter, so, what does "Make sure to avoid restarts" mean? Besides, is the parameter "1000" a reference value which will be changed in specific project in order to be converged? Or must use 1000 to get the condition number? Thanks. Jim >>? 2012-07-11 22:19:51?"Jed Brown" ??? >>On Wed, Jul 11, 2012 at 8:51 AM, w_ang_temp wrote: >>Hello, >> Is there function in PETSc for judging the properties of the matrix A ? Such as positive definitiveness and >>conditional number. I know that there are several functions begin with MatIs+, but it has limited amounts. >The Ritz values are estimates of the eigenvalues that you can obtain using KSPSolve() with ->ksp_compute_eigenvalues or -ksp_plot_eigenvalues. Note that these are estimates for the _preconditioned_ >operator, so you should use -pc_type none if you want estimates for the original operator. >http://www.mcs.anl.gov/petsc/documentation/faq.html#conditionnumber -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 7 08:59:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Aug 2012 08:59:05 -0500 Subject: [petsc-users] Functions for judging the properties of the matrix In-Reply-To: <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> References: <5daed011.ed92.138764ea964.Coremail.w_ang_temp@163.com> <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> Message-ID: On Tue, Aug 7, 2012 at 8:56 AM, w_ang_temp wrote: > > Hello, Jed > I am a little puzzled about the "How can I determine the condition > number of a matrix?" in the > FAQ. The details are as follows: > > "...For larger matrices, you can run with *-pc_type none -ksp_type > gmres -ksp_monitor_singular_value * > *-ksp_gmres_restart 1000* to get approximations to the condition number > of the operator. This will > generally be accurate for the largest singular values, but may > overestimate the smallest singular > value unless the method has converged. Make sure to avoid restarts...". > > Since the command "*-pc_type none -ksp_type gmres > -ksp_monitor_singular_value* > *-ksp_gmres_restart 1000*" using restart parameter, so, what does "Make > sure to avoid restarts" mean? > Do not believe the answer after the restart. > Besides, is the parameter "*1000*" a reference value which will be > changed in specific project in order > to be converged? Or must use *1000 *to get the condition number? > The former. Matt > Thanks. > Jim > > > > >>? 2012-07-11 22:19:51?"Jed Brown" ??? > > >>On Wed, Jul 11, 2012 at 8:51 AM, w_ang_temp wrote: > >> >> >>Hello, >> >> >> Is there function in PETSc for judging the properties of the matrix >> A ? Such as positive definitiveness and >> >> >>conditional number. I know that there are several functions begin with >> MatIs+, but it has limited amounts. >> > > >The Ritz values are estimates of the eigenvalues that you can obtain > using KSPSolve() with ->ksp_compute_eigenvalues or -ksp_plot_eigenvalues. > Note that these are estimates for the _preconditioned_ >operator, so you > should use -pc_type none if you want estimates for the original operator. > > >http://www.mcs.anl.gov/petsc/documentation/faq.html#conditionnumber > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Tue Aug 7 09:55:13 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Tue, 7 Aug 2012 22:55:13 +0800 (CST) Subject: [petsc-users] Functions for judging the properties of the matrix In-Reply-To: References: <5daed011.ed92.138764ea964.Coremail.w_ang_temp@163.com> <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> Message-ID: <6e718039.11b07.13901951ee1.Coremail.w_ang_temp@163.com> Hello, I am sorry that I cannot understand "Make sure to avoid restarts". In the FAQ, it says that using the restarted GMRES method to get approximations to the condition number. So why saying "avoid restarts" in the next following? Besides, there are two questions about GMRES: (1) Is the "iterations count" of GMRES with restarted parameters which is outputted by PETSc meaningless? (2) Is there only restarted GMRES in PETSc? Thanks. Jim >>? 2012-08-07 21:59:05?"Matthew Knepley" ??? >>On Tue, Aug 7, 2012 at 8:56 AM, w_ang_temp wrote: >>Hello, Jed >> I am a little puzzled about the "How can I determine the condition number of a matrix?" in the >>FAQ. The details are as follows: >> "...For larger matrices, you can run with -pc_type none -ksp_type gmres -ksp_monitor_singular_value >>-ksp_gmres_restart 1000 to get approximations to the condition number of the operator. This will >>generally be accurate for the largest singular values, but may overestimate the smallest singular >>value unless the method has converged. Make sure to avoid restarts...". >> Since the command "-pc_type none -ksp_type gmres -ksp_monitor_singular_value >>-ksp_gmres_restart 1000" using restart parameter, so, what does "Make sure to avoid restarts" mean? >Do not believe the answer after the restart. >>Besides, is the parameter "1000" a reference value which will be changed in specific project in order >>to be converged? Or must use 1000 to get the condition number? >The former. > Matt >> Thanks. >> Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 7 09:59:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Aug 2012 09:59:57 -0500 Subject: [petsc-users] Functions for judging the properties of the matrix In-Reply-To: <6e718039.11b07.13901951ee1.Coremail.w_ang_temp@163.com> References: <5daed011.ed92.138764ea964.Coremail.w_ang_temp@163.com> <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> <6e718039.11b07.13901951ee1.Coremail.w_ang_temp@163.com> Message-ID: On Tue, Aug 7, 2012 at 9:55 AM, w_ang_temp wrote: > Hello, > I am sorry that I cannot understand "Make sure to avoid restarts". In > the FAQ, it says that using the restarted > GMRES method to get approximations to the condition number. So why saying > "avoid restarts" in the next following? > Jim, in order to understand this, it is necessary to understand GMRES. Please read the section in Yousef Saad's book. We are factoring the Hessenberg matrix it produces to get estimates. If you restart, that is thrown away. Matt > Besides, there are two questions about GMRES: > (1) Is the "iterations count" of GMRES with restarted parameters which > is outputted by PETSc meaningless? > (2) Is there only restarted GMRES in PETSc? > Thanks. > Jim > > > >>? 2012-08-07 21:59:05?"Matthew Knepley" ??? > > >>On Tue, Aug 7, 2012 at 8:56 AM, w_ang_temp wrote: > >> >> >>Hello, Jed >> >> I am a little puzzled about the "How can I determine the condition >> number of a matrix?" in the >> >>FAQ. The details are as follows: >> >> >> "...For larger matrices, you can run with *-pc_type none -ksp_type >> gmres -ksp_monitor_singular_value * >> >>*-ksp_gmres_restart 1000* to get approximations to the condition >> number of the operator. This will >> >>generally be accurate for the largest singular values, but may >> overestimate the smallest singular >> >>value unless the method has converged. Make sure to avoid restarts...". >> >> >> Since the command "*-pc_type none -ksp_type gmres >> -ksp_monitor_singular_value* >> >>*-ksp_gmres_restart 1000*" using restart parameter, so, what does >> "Make sure to avoid restarts" mean? >> > > >Do not believe the answer after the restart. > > >> >>Besides, is the parameter "*1000*" a reference value which will be >> changed in specific project in order >> >>to be converged? Or must use *1000 *to get the condition number? >> > > >The former. > > > Matt > > >> >> Thanks. >> >> Jim >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Aug 7 10:05:19 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 7 Aug 2012 10:05:19 -0500 (CDT) Subject: [petsc-users] Functions for judging the properties of the matrix In-Reply-To: <6e718039.11b07.13901951ee1.Coremail.w_ang_temp@163.com> References: <5daed011.ed92.138764ea964.Coremail.w_ang_temp@163.com> <52c77c3d.13898.139015f6d1d.Coremail.w_ang_temp@163.com> <6e718039.11b07.13901951ee1.Coremail.w_ang_temp@163.com> Message-ID: For eg: -ksp_gmres_restart 1000 -ksp_max_it 1000 [default] avoids restarts But if your code doesn't converge before 1000 iterations - perhaps use: -ksp_gmres_restart 2000 -ksp_max_it 2000 Satish On Tue, 7 Aug 2012, w_ang_temp wrote: > Hello, > I am sorry that I cannot understand "Make sure to avoid restarts". In the FAQ, it says that using the restarted > GMRES method to get approximations to the condition number. So why saying "avoid restarts" in the next following? > Besides, there are two questions about GMRES: > (1) Is the "iterations count" of GMRES with restarted parameters which is outputted by PETSc meaningless? > (2) Is there only restarted GMRES in PETSc? > Thanks. > Jim > > > > > >>? 2012-08-07 21:59:05?"Matthew Knepley" ??? > >>On Tue, Aug 7, 2012 at 8:56 AM, w_ang_temp wrote: > > > >>Hello, Jed > >> I am a little puzzled about the "How can I determine the condition number of a matrix?" in the > >>FAQ. The details are as follows: > > >> "...For larger matrices, you can run with -pc_type none -ksp_type gmres -ksp_monitor_singular_value > >>-ksp_gmres_restart 1000 to get approximations to the condition number of the operator. This will > >>generally be accurate for the largest singular values, but may overestimate the smallest singular > >>value unless the method has converged. Make sure to avoid restarts...". > > >> Since the command "-pc_type none -ksp_type gmres -ksp_monitor_singular_value > >>-ksp_gmres_restart 1000" using restart parameter, so, what does "Make sure to avoid restarts" mean? > > > >Do not believe the answer after the restart. > > >>Besides, is the parameter "1000" a reference value which will be changed in specific project in order > >>to be converged? Or must use 1000 to get the condition number? > > > >The former. > > > > Matt > > >> Thanks. > >> Jim From stali at geology.wisc.edu Tue Aug 7 12:06:16 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Tue, 07 Aug 2012 12:06:16 -0500 Subject: [petsc-users] direct solver question Message-ID: <50214B08.3010703@geology.wisc.edu> Hello list By any chance is using -ksp_type gmres -pc_type lu similar to -ksp_type preonly -pc_type lu? Should the results be equivalent? T From bsmith at mcs.anl.gov Tue Aug 7 12:01:44 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Aug 2012 12:01:44 -0500 Subject: [petsc-users] direct solver question In-Reply-To: <50214B08.3010703@geology.wisc.edu> References: <50214B08.3010703@geology.wisc.edu> Message-ID: <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: > Hello list > > By any chance is using > > -ksp_type gmres -pc_type lu > > similar to > > -ksp_type preonly -pc_type lu? > > Should the results be equivalent? > > T The should be similar but will not be identical necessarily, in particular when the matrix is ill conditioned they can be very different and in fact GMRES may iterate a few times. Barry From stali at geology.wisc.edu Tue Aug 7 12:48:18 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Tue, 07 Aug 2012 12:48:18 -0500 Subject: [petsc-users] direct solver question In-Reply-To: <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> Message-ID: <502154E2.30409@geology.wisc.edu> Barry So which one of the two gives the correct solution? And yes the condition number is high. Actually I am trying to solve a trivial quasi-static problem with the following flags (1) -ksp_type gmres -pc_type asm (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero The results at t=0 for all three cases are equivalent. For the time dependent part (linear system is solved again), results from (1) and (2) are the same. But (3) gives a very different answer. I also solve the same quasi-static problem using a different code (does not use PETSc) and the results obtained are similar to (1) and (2). So basically I am trying to understand why '-ksp_type preonly' gives me a (potentially) wrong solution. Thanks Tabrez On 08/07/2012 12:01 PM, Barry Smith wrote: > On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: > >> Hello list >> >> By any chance is using >> >> -ksp_type gmres -pc_type lu >> >> similar to >> >> -ksp_type preonly -pc_type lu? >> >> Should the results be equivalent? >> >> T > The should be similar but will not be identical necessarily, in particular when the matrix is ill conditioned they can be very different and in fact GMRES may iterate a few times. > > Barry > -- No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley From knepley at gmail.com Tue Aug 7 13:05:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Aug 2012 13:05:01 -0500 Subject: [petsc-users] direct solver question In-Reply-To: <502154E2.30409@geology.wisc.edu> References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> <502154E2.30409@geology.wisc.edu> Message-ID: On Tue, Aug 7, 2012 at 12:48 PM, Tabrez Ali wrote: > Barry > > So which one of the two gives the correct solution? And yes the condition > number is high. > > Actually I am trying to solve a trivial quasi-static problem with the > following flags > > (1) -ksp_type gmres -pc_type asm > (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero > (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero > This does no iterative refinement, so you are stuck with the initial accuracy of the LU factorization, which can be bad for an ill-conditioned matrix. Matt > The results at t=0 for all three cases are equivalent. > > For the time dependent part (linear system is solved again), results from > (1) and (2) are the same. But (3) gives a very different answer. > > I also solve the same quasi-static problem using a different code (does > not use PETSc) and the results obtained are similar to (1) and (2). > > So basically I am trying to understand why '-ksp_type preonly' gives me a > (potentially) wrong solution. > > Thanks > Tabrez > > > > On 08/07/2012 12:01 PM, Barry Smith wrote: > >> On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: >> >> Hello list >>> >>> By any chance is using >>> >>> -ksp_type gmres -pc_type lu >>> >>> similar to >>> >>> -ksp_type preonly -pc_type lu? >>> >>> Should the results be equivalent? >>> >>> T >>> >> The should be similar but will not be identical necessarily, in >> particular when the matrix is ill conditioned they can be very different >> and in fact GMRES may iterate a few times. >> >> Barry >> >> > > -- > No one trusts a model except the one who wrote it; Everyone trusts an > observation except the one who made it- Harlow Shapley > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Tue Aug 7 13:53:57 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Tue, 07 Aug 2012 13:53:57 -0500 Subject: [petsc-users] direct solver question In-Reply-To: References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> <502154E2.30409@geology.wisc.edu> Message-ID: <50216445.2070207@geology.wisc.edu> Matt So if I want to use a direct solver I should really use (2) instead of (3). Tabrez On 08/07/2012 01:05 PM, Matthew Knepley wrote: > On Tue, Aug 7, 2012 at 12:48 PM, Tabrez Ali > wrote: > > Barry > > So which one of the two gives the correct solution? And yes the > condition number is high. > > Actually I am trying to solve a trivial quasi-static problem with > the following flags > > (1) -ksp_type gmres -pc_type asm > (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero > (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero > > > This does no iterative refinement, so you are stuck with the initial > accuracy of the LU > factorization, which can be bad for an ill-conditioned matrix. > > Matt > > The results at t=0 for all three cases are equivalent. > > For the time dependent part (linear system is solved again), > results from (1) and (2) are the same. But (3) gives a very > different answer. > > I also solve the same quasi-static problem using a different code > (does not use PETSc) and the results obtained are similar to (1) > and (2). > > So basically I am trying to understand why '-ksp_type preonly' > gives me a (potentially) wrong solution. > > Thanks > Tabrez > > > > On 08/07/2012 12:01 PM, Barry Smith wrote: > > On Aug 7, 2012, at 12:06 PM, Tabrez Ali > wrote: > > Hello list > > By any chance is using > > -ksp_type gmres -pc_type lu > > similar to > > -ksp_type preonly -pc_type lu? > > Should the results be equivalent? > > T > > The should be similar but will not be identical > necessarily, in particular when the matrix is ill conditioned > they can be very different and in fact GMRES may iterate a few > times. > > Barry > > > > -- > No one trusts a model except the one who wrote it; Everyone trusts > an observation except the one who made it- Harlow Shapley > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 7 13:59:25 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Aug 2012 13:59:25 -0500 Subject: [petsc-users] direct solver question In-Reply-To: <50216445.2070207@geology.wisc.edu> References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> <502154E2.30409@geology.wisc.edu> <50216445.2070207@geology.wisc.edu> Message-ID: On Tue, Aug 7, 2012 at 1:53 PM, Tabrez Ali wrote: > Matt > > So if I want to use a direct solver I should really use (2) instead of (3). > To guarantee robustness, yes. Matt > Tabrez > > On 08/07/2012 01:05 PM, Matthew Knepley wrote: > > On Tue, Aug 7, 2012 at 12:48 PM, Tabrez Ali wrote: > >> Barry >> >> So which one of the two gives the correct solution? And yes the condition >> number is high. >> >> Actually I am trying to solve a trivial quasi-static problem with the >> following flags >> >> (1) -ksp_type gmres -pc_type asm >> (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero >> (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero >> > > This does no iterative refinement, so you are stuck with the initial > accuracy of the LU > factorization, which can be bad for an ill-conditioned matrix. > > Matt > > >> The results at t=0 for all three cases are equivalent. >> >> For the time dependent part (linear system is solved again), results from >> (1) and (2) are the same. But (3) gives a very different answer. >> >> I also solve the same quasi-static problem using a different code (does >> not use PETSc) and the results obtained are similar to (1) and (2). >> >> So basically I am trying to understand why '-ksp_type preonly' gives me a >> (potentially) wrong solution. >> >> Thanks >> Tabrez >> >> >> >> On 08/07/2012 12:01 PM, Barry Smith wrote: >> >>> On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: >>> >>> Hello list >>>> >>>> By any chance is using >>>> >>>> -ksp_type gmres -pc_type lu >>>> >>>> similar to >>>> >>>> -ksp_type preonly -pc_type lu? >>>> >>>> Should the results be equivalent? >>>> >>>> T >>>> >>> The should be similar but will not be identical necessarily, in >>> particular when the matrix is ill conditioned they can be very different >>> and in fact GMRES may iterate a few times. >>> >>> Barry >>> >>> >> >> -- >> No one trusts a model except the one who wrote it; Everyone trusts an >> observation except the one who made it- Harlow Shapley >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 7 14:39:51 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Aug 2012 14:39:51 -0500 Subject: [petsc-users] direct solver question In-Reply-To: References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> <502154E2.30409@geology.wisc.edu> <50216445.2070207@geology.wisc.edu> Message-ID: On Aug 7, 2012, at 1:59 PM, Matthew Knepley wrote: > On Tue, Aug 7, 2012 at 1:53 PM, Tabrez Ali wrote: > Matt > > So if I want to use a direct solver I should really use (2) instead of (3). > > To guarantee robustness, yes. Note also the -pc_factor_shift_type nonzero if this truly is needed then that means A != LU but instead A + dI = LU for some d > 0 which means the LU factorization is not for A and hence a single solve of the LU will not solver for A and it needs to iterate hence you need the GMRES. Barry > > Matt > > Tabrez > > On 08/07/2012 01:05 PM, Matthew Knepley wrote: >> On Tue, Aug 7, 2012 at 12:48 PM, Tabrez Ali wrote: >> Barry >> >> So which one of the two gives the correct solution? And yes the condition number is high. >> >> Actually I am trying to solve a trivial quasi-static problem with the following flags >> >> (1) -ksp_type gmres -pc_type asm >> (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero >> (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero >> >> This does no iterative refinement, so you are stuck with the initial accuracy of the LU >> factorization, which can be bad for an ill-conditioned matrix. >> >> Matt >> >> The results at t=0 for all three cases are equivalent. >> >> For the time dependent part (linear system is solved again), results from (1) and (2) are the same. But (3) gives a very different answer. >> >> I also solve the same quasi-static problem using a different code (does not use PETSc) and the results obtained are similar to (1) and (2). >> >> So basically I am trying to understand why '-ksp_type preonly' gives me a (potentially) wrong solution. >> >> Thanks >> Tabrez >> >> >> >> On 08/07/2012 12:01 PM, Barry Smith wrote: >> On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: >> >> Hello list >> >> By any chance is using >> >> -ksp_type gmres -pc_type lu >> >> similar to >> >> -ksp_type preonly -pc_type lu? >> >> Should the results be equivalent? >> >> T >> The should be similar but will not be identical necessarily, in particular when the matrix is ill conditioned they can be very different and in fact GMRES may iterate a few times. >> >> Barry >> >> >> >> -- >> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From stali at geology.wisc.edu Tue Aug 7 15:37:00 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Tue, 07 Aug 2012 15:37:00 -0500 Subject: [petsc-users] direct solver question In-Reply-To: References: <50214B08.3010703@geology.wisc.edu> <52D65243-C440-475F-A944-B86BB02D510C@mcs.anl.gov> <502154E2.30409@geology.wisc.edu> <50216445.2070207@geology.wisc.edu> Message-ID: <50217C6C.4040201@geology.wisc.edu> Thanks everyone! It now makes sense. Tabrez On 08/07/2012 02:39 PM, Barry Smith wrote: > On Aug 7, 2012, at 1:59 PM, Matthew Knepley wrote: > >> On Tue, Aug 7, 2012 at 1:53 PM, Tabrez Ali wrote: >> Matt >> >> So if I want to use a direct solver I should really use (2) instead of (3). >> >> To guarantee robustness, yes. > Note also the -pc_factor_shift_type nonzero if this truly is needed then that means A != LU but instead A + dI = LU for some d> 0 which means the LU factorization is not for A and hence a single solve of the LU will not solver for A and it needs to iterate hence you need the GMRES. > > Barry > >> Matt >> >> Tabrez >> >> On 08/07/2012 01:05 PM, Matthew Knepley wrote: >>> On Tue, Aug 7, 2012 at 12:48 PM, Tabrez Ali wrote: >>> Barry >>> >>> So which one of the two gives the correct solution? And yes the condition number is high. >>> >>> Actually I am trying to solve a trivial quasi-static problem with the following flags >>> >>> (1) -ksp_type gmres -pc_type asm >>> (2) -ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero >>> (3) -ksp_type preonly -pc_type lu -pc_factor_shift_type nonzero >>> >>> This does no iterative refinement, so you are stuck with the initial accuracy of the LU >>> factorization, which can be bad for an ill-conditioned matrix. >>> >>> Matt >>> >>> The results at t=0 for all three cases are equivalent. >>> >>> For the time dependent part (linear system is solved again), results from (1) and (2) are the same. But (3) gives a very different answer. >>> >>> I also solve the same quasi-static problem using a different code (does not use PETSc) and the results obtained are similar to (1) and (2). >>> >>> So basically I am trying to understand why '-ksp_type preonly' gives me a (potentially) wrong solution. >>> >>> Thanks >>> Tabrez >>> >>> >>> >>> On 08/07/2012 12:01 PM, Barry Smith wrote: >>> On Aug 7, 2012, at 12:06 PM, Tabrez Ali wrote: >>> >>> Hello list >>> >>> By any chance is using >>> >>> -ksp_type gmres -pc_type lu >>> >>> similar to >>> >>> -ksp_type preonly -pc_type lu? >>> >>> Should the results be equivalent? >>> >>> T >>> The should be similar but will not be identical necessarily, in particular when the matrix is ill conditioned they can be very different and in fact GMRES may iterate a few times. >>> >>> Barry >>> >>> >>> >>> -- >>> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener From lzj at umich.edu Tue Aug 7 16:28:27 2012 From: lzj at umich.edu (Peter Zhoujie Lu) Date: Tue, 7 Aug 2012 17:28:27 -0400 Subject: [petsc-users] petscvec.mod version problem In-Reply-To: References: Message-ID: Hi All, I am trying to compile my code that uses petscvec, I got an error about version of petscmod.mod. I tried to search online, but couldn't find the answer. I was wondering if anyone can help. I am using gfortran, openmpi on OSX 10.7. The petsc version is 3.3-p2. Thanks. Peter openmpif90 -I../../mod -I../../externals/SU_MPI/mod -I../../externals/ADT/mod -I../../src/python/fortran/aeroElastic -I../../src/inputParam -I../../src/turbulence -I/opt/local/include -DHAS_ISNAN -fpic -r8 -O2 -g -I/Users/lvzhoujie/Work/petsc-3.3-p2/include -I/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/include -I/Users/lvzhoujie/Work/petsc-3.3-p2 -c getADjoint.F90 -o ../../obj/getADjoint.o getADjoint.F90:18.14: use petscvec 1 Fatal Error: Wrong module version '9' (expected '4') for file 'petscvec.mod' opened at (1) make[4]: *** [getADjoint.o] Error 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Aug 7 16:30:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 7 Aug 2012 15:30:42 -0600 Subject: [petsc-users] petscvec.mod version problem In-Reply-To: References: Message-ID: There is a version of the module compiled using an older compiler version. It could come from here or from system paths. You have to track down the stray module file and delete it (or at least get it out of a path searched by the compiler). On Tue, Aug 7, 2012 at 3:28 PM, Peter Zhoujie Lu wrote: > Hi All, > > I am trying to compile my code that uses petscvec, I got an error about > version of petscmod.mod. I tried to search online, but couldn't find the > answer. I was wondering if anyone can help. I am using gfortran, openmpi on > OSX 10.7. The petsc version is 3.3-p2. Thanks. > > Peter > > openmpif90 -I../../mod -I../../externals/SU_MPI/mod > -I../../externals/ADT/mod -I../../src/python/fortran/aeroElastic > -I../../src/inputParam -I../../src/turbulence -I/opt/local/include > -DHAS_ISNAN -fpic -r8 -O2 -g > -I/Users/lvzhoujie/Work/petsc-3.3-p2/include > -I/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/include > -I/Users/lvzhoujie/Work/petsc-3.3-p2 -c getADjoint.F90 -o > ../../obj/getADjoint.o > > getADjoint.F90:18.14: > > use petscvec > 1 > Fatal Error: Wrong module version '9' (expected '4') for file > 'petscvec.mod' opened at (1) > make[4]: *** [getADjoint.o] Error 1 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzj at umich.edu Tue Aug 7 16:40:57 2012 From: lzj at umich.edu (Peter Zhoujie Lu) Date: Tue, 7 Aug 2012 17:40:57 -0400 Subject: [petsc-users] petscvec.mod version problem In-Reply-To: References: Message-ID: Hi Jed, Thanks for your reply. Sorry I am not quite sure about what you meant. Did you mean that petscvec.mod was compiled with a older compiler? How to track down this? I compiled petsc with gcc(4.8). Peter On Tue, Aug 7, 2012 at 5:30 PM, Jed Brown wrote: > There is a version of the module compiled using an older compiler version. > It could come from here or from system paths. You have to track down the > stray module file and delete it (or at least get it out of a path searched > by the compiler). > > > On Tue, Aug 7, 2012 at 3:28 PM, Peter Zhoujie Lu wrote: > >> Hi All, >> >> I am trying to compile my code that uses petscvec, I got an error about >> version of petscmod.mod. I tried to search online, but couldn't find the >> answer. I was wondering if anyone can help. I am using gfortran, openmpi on >> OSX 10.7. The petsc version is 3.3-p2. Thanks. >> >> Peter >> >> openmpif90 -I../../mod -I../../externals/SU_MPI/mod >> -I../../externals/ADT/mod -I../../src/python/fortran/aeroElastic >> -I../../src/inputParam -I../../src/turbulence -I/opt/local/include >> -DHAS_ISNAN -fpic -r8 -O2 -g >> -I/Users/lvzhoujie/Work/petsc-3.3-p2/include >> -I/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/include >> -I/Users/lvzhoujie/Work/petsc-3.3-p2 -c getADjoint.F90 -o >> ../../obj/getADjoint.o >> >> getADjoint.F90:18.14: >> >> use petscvec >> 1 >> Fatal Error: Wrong module version '9' (expected '4') for file >> 'petscvec.mod' opened at (1) >> make[4]: *** [getADjoint.o] Error 1 >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Aug 7 16:47:33 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 7 Aug 2012 16:47:33 -0500 (CDT) Subject: [petsc-users] petscvec.mod version problem In-Reply-To: References: Message-ID: 1. Can you reproduce this with a petsc example? i.e src/vec/vec/examples/tutorials/ex20f90 2. Can you confirm openmpif90 is built with the same gfortran 4.8? Perhaps the usage of -r8 is causing this error? Satish On Tue, 7 Aug 2012, Peter Zhoujie Lu wrote: > Hi Jed, > > Thanks for your reply. Sorry I am not quite sure about what you meant. Did > you mean that petscvec.mod was compiled with a older compiler? How to track > down this? I compiled petsc with gcc(4.8). > > Peter > > On Tue, Aug 7, 2012 at 5:30 PM, Jed Brown wrote: > > > There is a version of the module compiled using an older compiler version. > > It could come from here or from system paths. You have to track down the > > stray module file and delete it (or at least get it out of a path searched > > by the compiler). > > > > > > On Tue, Aug 7, 2012 at 3:28 PM, Peter Zhoujie Lu wrote: > > > >> Hi All, > >> > >> I am trying to compile my code that uses petscvec, I got an error about > >> version of petscmod.mod. I tried to search online, but couldn't find the > >> answer. I was wondering if anyone can help. I am using gfortran, openmpi on > >> OSX 10.7. The petsc version is 3.3-p2. Thanks. > >> > >> Peter > >> > >> openmpif90 -I../../mod -I../../externals/SU_MPI/mod > >> -I../../externals/ADT/mod -I../../src/python/fortran/aeroElastic > >> -I../../src/inputParam -I../../src/turbulence -I/opt/local/include > >> -DHAS_ISNAN -fpic -r8 -O2 -g > >> -I/Users/lvzhoujie/Work/petsc-3.3-p2/include > >> -I/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/include > >> -I/Users/lvzhoujie/Work/petsc-3.3-p2 -c getADjoint.F90 -o > >> ../../obj/getADjoint.o > >> > >> getADjoint.F90:18.14: > >> > >> use petscvec > >> 1 > >> Fatal Error: Wrong module version '9' (expected '4') for file > >> 'petscvec.mod' opened at (1) > >> make[4]: *** [getADjoint.o] Error 1 > >> > >> > > > From bsmith at mcs.anl.gov Tue Aug 7 20:18:58 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Aug 2012 20:18:58 -0500 Subject: [petsc-users] AMS parallel usage In-Reply-To: <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> Message-ID: I'll take a look at this but it may take a little time. It could be some of the examples are out of date. Barry On Aug 7, 2012, at 5:25 AM, Gong Ding wrote: > Some more bugs about AMS_Comm_publish for parallel situation. even the AMS mpi example can not work. > > 1) parameter mismatch > asmpub.c line 88 > mysetport = *(int *) va_arg(args, int *); > It seems author wants to add another argument as int * to AMS_Comm_publish function for parallel situation. which not appears in the manual. > I just comment line 88 and line 130-131, > if (mysetport != -1) { > AMS_port = mysetport; > } > > 2) extra check of port number > asmpub.c line 149 > /* Create publisher arguments */ > err = AMSP_New_Pub_Arguments(AMS_port, &pub_arg); > CheckErr(err); > > AMSP_New_Pub_Arguments function will check the value of AMS_port. A port number of -1 or 0 will return a error. > > > 3) AMSP_start_publisher_func seems require a port number -1 to force socket bind function to allocate a new port, which conflict with AMSP_New_Pub_Arguments check. > > nettcppub.c line 66 > if (*port == -1) > *port = 0; > at line 94, a bind function is called. > > > I have to set pub_arg->port = -1 as follows. > > if(ctype == MPI_TYPE) > { > pub_arg->care_about_port = 0; > pub_arg->port = -1; > } > else > pub_arg->care_about_port = 1; > > > Barry, will you please check it? > > >> >> >> On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: >> >> >> >>> Thanks, Barry. It is really a wonderful tool. >> >> >> >> Thanks >> >> >> >>> PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. >> >>> >> >>> Hope the AMS manual have some more detailed description to parallel usage and data types. >> >> >> >> Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with >> >> >> >> hg clone http://petsc.cs.iit.edu/petsc/ams-dev >> >> >> >> >> >> Barry >> >> >> >>> >> >>> >> >>>> >> >>>> >> >>>> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: >> >>>> >> >>>> >> >>>> >> >>>>> Hi, >> >>>> >> >>>>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. >> >>>> >> >>>> >> >>>> >> >>>> You need to build AMS with the same MPI that you will use in the parallel program. >> >>>> >> >>>> >> >>>> >> >>>>> >> >>>> >> >>>>> I'd like to know some thing about data synchronization. >> >>>> >> >>>>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? >> >>>> >> >>>> >> >>>> >> >>>> There are different types of variables you can publish. >> >>>> >> >>>> >> >>>> >> >>>> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. >> >>>> >> >>>> >> >>>> >> >>>> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . >> >>>> >> >>>> >> >>>> >> >>>> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. >> >>>> >> >>>> >> >>>> >> >>>>> >> >>>> >> >>>>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? >> >>>> >> >>>> >> >>>> >> >>>> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. >> >>>> >> >>>> >> >>>> >> >>>>> >> >>>> >> >>>>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? >> >>>> >> >>>> >> >>>> >> >>>> Nope, you do not need to do this. >> >>>> >> >>>> >> >>>> >> >>>> Barry >> >>>> >> >>>> >> >>>> >> >>>>> >> >>>> >> >>>>> Gong Ding >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >> >> >> From gdiso at ustc.edu Tue Aug 7 21:11:47 2012 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 8 Aug 2012 10:11:47 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> Message-ID: <12546624.278481344391907353.JavaMail.coremail@mail.ustc.edu> Hope you can do this in a few days. My project has a deadline of Oct, 1. > > > I'll take a look at this but it may take a little time. It could be some of the examples are out of date. > > > > > > Barry > > > > On Aug 7, 2012, at 5:25 AM, Gong Ding wrote: > > > > > Some more bugs about AMS_Comm_publish for parallel situation. even the AMS mpi example can not work. > > > > > > 1) parameter mismatch > > > asmpub.c line 88 > > > mysetport = *(int *) va_arg(args, int *); > > > It seems author wants to add another argument as int * to AMS_Comm_publish function for parallel situation. which not appears in the manual. > > > I just comment line 88 and line 130-131, > > > if (mysetport != -1) { > > > AMS_port = mysetport; > > > } > > > > > > 2) extra check of port number > > > asmpub.c line 149 > > > /* Create publisher arguments */ > > > err = AMSP_New_Pub_Arguments(AMS_port, &pub_arg); > > > CheckErr(err); > > > > > > AMSP_New_Pub_Arguments function will check the value of AMS_port. A port number of -1 or 0 will return a error. > > > > > > > > > 3) AMSP_start_publisher_func seems require a port number -1 to force socket bind function to allocate a new port, which conflict with AMSP_New_Pub_Arguments check. > > > > > > nettcppub.c line 66 > > > if (*port == -1) > > > *port = 0; > > > at line 94, a bind function is called. > > > > > > > > > I have to set pub_arg->port = -1 as follows. > > > > > > if(ctype == MPI_TYPE) > > > { > > > pub_arg->care_about_port = 0; > > > pub_arg->port = -1; > > > } > > > else > > > pub_arg->care_about_port = 1; > > > > > > > > > Barry, will you please check it? > > > > > > > > >> > > >> > > >> On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: > > >> > > >> > > >> > > >>> Thanks, Barry. It is really a wonderful tool. > > >> > > >> > > >> > > >> Thanks > > >> > > >> > > >> > > >>> PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. > > >> > > >>> > > >> > > >>> Hope the AMS manual have some more detailed description to parallel usage and data types. > > >> > > >> > > >> > > >> Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with > > >> > > >> > > >> > > >> hg clone http://petsc.cs.iit.edu/petsc/ams-dev > > >> > > >> > > >> > > >> > > >> > > >> Barry > > >> > > >> > > >> > > >>> > > >> > > >>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> Hi, > > >> > > >>>> > > >> > > >>>>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> You need to build AMS with the same MPI that you will use in the parallel program. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> I'd like to know some thing about data synchronization. > > >> > > >>>> > > >> > > >>>>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> There are different types of variables you can publish. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Nope, you do not need to do this. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Barry > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> Gong Ding > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >> > > >> > > >> > > > > From gdiso at ustc.edu Tue Aug 7 22:16:36 2012 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 8 Aug 2012 11:16:36 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> Message-ID: <9805567.278611344395796516.JavaMail.coremail@mail.ustc.edu> Finally, the port conflict was solved by following c code in asmpub.c around line 158. The rank 0 processor use default port, others use a port specified by os. if(ctype == MPI_TYPE) { if(rank == 0) { pub_arg->care_about_port = 1; } else { pub_arg->care_about_port = 0; pub_arg->port = -1; } } else pub_arg->care_about_port = 1; Now AMS works well for parallel environment. > > I'll take a look at this but it may take a little time. It could be some of the examples are out of date. > > > > > > Barry > > > > On Aug 7, 2012, at 5:25 AM, Gong Ding wrote: > > > > > Some more bugs about AMS_Comm_publish for parallel situation. even the AMS mpi example can not work. > > > > > > 1) parameter mismatch > > > asmpub.c line 88 > > > mysetport = *(int *) va_arg(args, int *); > > > It seems author wants to add another argument as int * to AMS_Comm_publish function for parallel situation. which not appears in the manual. > > > I just comment line 88 and line 130-131, > > > if (mysetport != -1) { > > > AMS_port = mysetport; > > > } > > > > > > 2) extra check of port number > > > asmpub.c line 149 > > > /* Create publisher arguments */ > > > err = AMSP_New_Pub_Arguments(AMS_port, &pub_arg); > > > CheckErr(err); > > > > > > AMSP_New_Pub_Arguments function will check the value of AMS_port. A port number of -1 or 0 will return a error. > > > > > > > > > 3) AMSP_start_publisher_func seems require a port number -1 to force socket bind function to allocate a new port, which conflict with AMSP_New_Pub_Arguments check. > > > > > > nettcppub.c line 66 > > > if (*port == -1) > > > *port = 0; > > > at line 94, a bind function is called. > > > > > > > > > I have to set pub_arg->port = -1 as follows. > > > > > > if(ctype == MPI_TYPE) > > > { > > > pub_arg->care_about_port = 0; > > > pub_arg->port = -1; > > > } > > > else > > > pub_arg->care_about_port = 1; > > > > > > > > > Barry, will you please check it? > > > > > > > > >> > > >> > > >> On Aug 6, 2012, at 10:17 PM, "Gong Ding" wrote: > > >> > > >> > > >> > > >>> Thanks, Barry. It is really a wonderful tool. > > >> > > >> > > >> > > >> Thanks > > >> > > >> > > >> > > >>> PS: The C STRING support of AMS is a bit strange. At very beginning, I can't understand the mechanism of STRING. So I added a CHAR data type to AMS for an alternate of STRING. > > >> > > >>> > > >> > > >>> Hope the AMS manual have some more detailed description to parallel usage and data types. > > >> > > >> > > >> > > >> Maybe you can help write it. We really don't have any one maintaining/improving the AMS which is a pity. You can access the repository with > > >> > > >> > > >> > > >> hg clone http://petsc.cs.iit.edu/petsc/ams-dev > > >> > > >> > > >> > > >> > > >> > > >> Barry > > >> > > >> > > >> > > >>> > > >> > > >>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> On Aug 6, 2012, at 9:36 AM, "Gong Ding" wrote: > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> Hi, > > >> > > >>>> > > >> > > >>>>> I am trying to use AMS as data communicator between computational code and user monitor. The AMS works well for serial situation, however, I don't know how to use it for parallel environment. The manual has little information, too. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> You need to build AMS with the same MPI that you will use in the parallel program. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> I'd like to know some thing about data synchronization. > > >> > > >>>> > > >> > > >>>>> Does AMS process data synchronization in MPI environment automatically? Should I call AMS_Comm_publish on every MPI processor, and create memory with AMS_Memory_create for my variables on each processor? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> There are different types of variables you can publish. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS_Comm_publish() takes an argument AMS_Comm_type which would be MPI_TYPE like in your case. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS_Memory_add_field() takes an argument AMS_Shared_type which can be AMS_COMMON, AMS_DISTRIBUTED or AMS_REDUCED and if it is reduced the AMS_Reduction_type is AMS_SUM, AMS_MAX, AMS_MIN, or AMS_NONE. See the manual page for AMS_Memory_add_field() . > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> If AMS_Shared_type is AMS_COMMON then AMS assumes that the values in those locations are identical on all processes (and it brings over to the accessor one set of those values), if distributed then each process has an array of different values (and it brings over all those values, in a long array, on the accessor) if reduce that it brings back the values after applying the reduction operator across all the processes; so with sum it beings back the sum of the values over all the processes. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> If so, the accessor will see how many TCP/IP port? Or accessor only communicate with MPI processor 0 and processor 0 will broadcast the information to all the others? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> AMS completely manages the fact there are multiple "publishers", the accessor transparently handles getting stuff from all the publisher nodes. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> If not, should I only create AMS object on processor 0, and processor 0 should broadcast what AMS get? > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Nope, you do not need to do this. > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> Barry > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> Gong Ding > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >>>> > > >> > > >> > > >> > > >> > > > > From gdiso at ustc.edu Tue Aug 7 22:30:18 2012 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 8 Aug 2012 11:30:18 +0800 (CST) Subject: [petsc-users] AMS parallel usage In-Reply-To: References: <538FE22F-2BC2-43E3-BA83-C663B9B9EE36@mcs.anl.gov> <13624031.277051344263773795.JavaMail.coremail@mail.ustc.edu> <5206031.277421344309466145.JavaMail.coremail@mail.ustc.edu> <27204485.277791344335115590.JavaMail.coremail@mail.ustc.edu> Message-ID: <22830901.278661344396618805.JavaMail.coremail@mail.ustc.edu> patch file for amspub.c -------------- next part -------------- A non-text attachment was scrubbed... Name: patch Type: application/octet-stream Size: 1977 bytes Desc: not available URL: From t.hisch at gmail.com Wed Aug 8 04:52:34 2012 From: t.hisch at gmail.com (Thomas Hisch) Date: Wed, 8 Aug 2012 11:52:34 +0200 Subject: [petsc-users] Speedup calculation of a high number of generalized eigenvalue problems Message-ID: Hello list, What can be done to speedup the calculation of the eigenvals and eigenvecs of many (~1000)!! generalized EVP, where the operator A is constant for all GEVPs and only B is different (but the nonzero pattern, size, type,.. are the same for all B). For now I create an EPS object for every pair (A,B) like in the following code: (Note: B is updated in other functions) void SlepcEigenSolver::Solve(ngbla::Matrix &evecs, ngbla::Vector &evals) { .... Vec xev; MatGetVecs(A_,PETSC_NULL,&xev); PetscPrintf(PETSC_COMM_WORLD,"Create EPS object and solve eigensystem\n"); EPSCreate(PETSC_COMM_WORLD, &eps_); EPSSetOperators(eps_,A_,B_); EPSSetDimensions(eps_, GetNev(), PETSC_DECIDE, PETSC_DECIDE); EPSSetFromOptions(eps_); PetscScalar target = GetShift(); target *= target; EPSSetTarget(eps_,target); EPSSetWhichEigenpairs(eps_,EPS_TARGET_MAGNITUDE); ST myst; EPSGetST(eps_,&myst); STSetType(myst,STSINVERT); PetscLogDouble t1,t2,t3; PetscGetTime(&t1); EPSSetUp(eps_); PetscGetTime(&t2); EPSSolve(eps_); PetscGetTime(&t3); PetscPrintf(PETSC_COMM_WORLD," -> EPS Solving Done\n\n"); PetscPrintf(PETSC_COMM_WORLD,"EPS: Elapsed time: %G (setup), %G (solve)\n",t2-t1,t3-t2); PetscInt its; EPSGetIterationNumber(eps_,&its); PetscPrintf(PETSC_COMM_WORLD, "number of iterations of the method %D\n", its); PetscInt nconv; EPSGetConverged(eps_,&nconv); PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %D\n\n",nconv); if (nconv>0) { evals.SetSize(nconv); if(!rank_) { evecs.SetSize(matsize,nconv); } PetscScalar lambda; // = k PetscReal error,re,im; for (PetscInt i=0;i(tmpptr); for(size_t j = 0; j < matsize; j++) { evecs(j,i) = evecptr[j]; } } VecRestoreArray(xev, &tmpptr); // destroy scatter context and local vector when no longer needed VecScatterDestroy(&ctx); VecDestroy(&vout); } /* end ugly inefficient(?) vector gather code */ evals(i) = static_cast(lambda); } PetscPrintf(PETSC_COMM_WORLD,"\n"); } VecDestroy(&xev); EPSDestroy(&eps_); if(!nconv) return; if(rank_) evecs.SetSize(matsize,nconv); //every process already owns all the eigenvalues //but not all eigenstates -> only bcast all eigenstates MPI_Bcast(&evecs(0), nconv*matsize, MPI_DOUBLE_COMPLEX, 0, MPI_COMM_WORLD); } Here are some facts about my EVP: The dimension of the Operators A & B is ~90000. For nev=1 the setup time (EPSSetUp) is 18sec, and the solve time 11sec. (on 16 nodes) -> number of iterations of the method 2 number of converged eigenpairs: 6 For nev=150 the setup time (EPSSetUp) is 18sec, and the solve time 65sec. (on 16 nodes) -> number of iterations of the method 1 number of converged eigenpairs: 155 (The output of -log_summary is attached) Do you think that there is any benefit in creating the EPS object at the beginning of my program and only call EPSSetOperators in my Solve function? Regards Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: out_nodebugging_full.log Type: application/octet-stream Size: 11312 bytes Desc: not available URL: From jroman at dsic.upv.es Wed Aug 8 06:43:57 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Aug 2012 13:43:57 +0200 Subject: [petsc-users] Speedup calculation of a high number of generalized eigenvalue problems In-Reply-To: References: Message-ID: <18E86B24-AB21-472C-B8F1-6577D0D3FC69@dsic.upv.es> El 08/08/2012, a las 11:52, Thomas Hisch escribi?: > Hello list, > > What can be done to speedup the calculation of the eigenvals and > eigenvecs of many (~1000)!! generalized EVP, where the operator A is > constant for all GEVPs and only B is different (but the nonzero > pattern, size, type,.. are the same for all B). For now I create an > EPS object for every pair (A,B) like in the following code: (Note: B > is updated in other functions) > > > void SlepcEigenSolver::Solve(ngbla::Matrix &evecs, > ngbla::Vector &evals) > { > .... > > Vec xev; > MatGetVecs(A_,PETSC_NULL,&xev); > PetscPrintf(PETSC_COMM_WORLD,"Create EPS object and solve eigensystem\n"); > EPSCreate(PETSC_COMM_WORLD, &eps_); > EPSSetOperators(eps_,A_,B_); > EPSSetDimensions(eps_, GetNev(), PETSC_DECIDE, PETSC_DECIDE); > EPSSetFromOptions(eps_); > > PetscScalar target = GetShift(); > target *= target; > > EPSSetTarget(eps_,target); > EPSSetWhichEigenpairs(eps_,EPS_TARGET_MAGNITUDE); > > ST myst; > EPSGetST(eps_,&myst); > STSetType(myst,STSINVERT); > > PetscLogDouble t1,t2,t3; > PetscGetTime(&t1); > EPSSetUp(eps_); > PetscGetTime(&t2); > EPSSolve(eps_); > PetscGetTime(&t3); > PetscPrintf(PETSC_COMM_WORLD," -> EPS Solving Done\n\n"); > PetscPrintf(PETSC_COMM_WORLD,"EPS: Elapsed time: %G (setup), %G > (solve)\n",t2-t1,t3-t2); > > PetscInt its; > EPSGetIterationNumber(eps_,&its); > > PetscPrintf(PETSC_COMM_WORLD, "number of iterations of the method %D\n", its); > > PetscInt nconv; > EPSGetConverged(eps_,&nconv); > PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %D\n\n",nconv); > > if (nconv>0) { > evals.SetSize(nconv); > if(!rank_) { > evecs.SetSize(matsize,nconv); > } > > PetscScalar lambda; // = k > PetscReal error,re,im; > > for (PetscInt i=0;i // Get converged eigenpairs: i-th eigenvalue is stored in kr > (real part) and > // ki (imaginary part) > EPSGetEigenpair(eps_,i,&lambda,PETSC_NULL,xev,PETSC_NULL); > > PetscScalar* tmpptr; > > // begin ugly inefficient(?) eigenvector gather code > { > VecScatter ctx; > Vec vout; > VecScatterCreateToZero(xev,&ctx,&vout); > // scatter as many times as you need > VecScatterBegin(ctx,xev,vout,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,xev,vout,INSERT_VALUES,SCATTER_FORWARD); > > int locsize; > VecGetLocalSize(vout,&locsize); > VecGetArray(vout,&tmpptr); > if(!rank_) { > const ngla::Complex *evecptr = static_cast(tmpptr); > > for(size_t j = 0; j < matsize; j++) { > evecs(j,i) = evecptr[j]; > } > } > > VecRestoreArray(xev, &tmpptr); > // destroy scatter context and local vector when no longer needed > VecScatterDestroy(&ctx); > VecDestroy(&vout); > } > /* end ugly inefficient(?) vector gather code */ > > evals(i) = static_cast(lambda); > } > PetscPrintf(PETSC_COMM_WORLD,"\n"); > } > > VecDestroy(&xev); > EPSDestroy(&eps_); > > if(!nconv) > return; > > if(rank_) > evecs.SetSize(matsize,nconv); > > //every process already owns all the eigenvalues > //but not all eigenstates -> only bcast all eigenstates > MPI_Bcast(&evecs(0), nconv*matsize, MPI_DOUBLE_COMPLEX, 0, MPI_COMM_WORLD); > } > > Here are some facts about my EVP: > The dimension of the Operators A & B is ~90000. > For nev=1 the setup time (EPSSetUp) is 18sec, and the solve time > 11sec. (on 16 nodes) > -> number of iterations of the method 2 > number of converged eigenpairs: 6 > > For nev=150 the setup time (EPSSetUp) is 18sec, and the solve time > 65sec. (on 16 nodes) > -> number of iterations of the method 1 > number of converged eigenpairs: 155 > (The output of -log_summary is attached) > > > Do you think that there is any benefit in creating the EPS object at > the beginning of my program and only call EPSSetOperators in my Solve > function? > > Regards > Thomas > Yes, you can get some modest benefit. I would strongly suggest using a parallel linear solver such as MUMPS. See section 3.4.1 of SLEPc's manual. Jose From ccchiou2012 at gmail.com Wed Aug 8 02:30:52 2012 From: ccchiou2012 at gmail.com (C.-C. Chiou) Date: Wed, 8 Aug 2012 15:30:52 +0800 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure Message-ID: Dear all, For concerning more general geometry of physical domain, for some situations, we need to apply the multi-block mesh. The attached figure (or alternatively the following link) illustrates one of examples. https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg Have you ever dealt with such kind of case with application of the PETSc object 'DA' ? Suppose we build multiple DA(or DM) for governing the multi-block data structure, in other words, we respectively build a DA for each block of mesh. Especially in parallel computing, how can we manage and control the communications between blocks of data accurately and efficiently? And, how can we ensure the correctness of parallel computational results? I wonder whether some possible application of PETSc object 'DA' (or DM) is sufficient for dealing with cases in multi-block mesh. Sincerely regards. Chuei-Ching Chiou -- Chuei-Ching Chiou, Ph. D student, Aerothermal & Plasma Physics Laboratory Department of Mechanical Engineering National Chiao Tung University Hsinchu 30010, TAIWAN TEL: +886-35712121 ext. 55175 E-mail: C CChiou2012 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: IMAG0003.jpg Type: image/jpeg Size: 801591 bytes Desc: not available URL: From knepley at gmail.com Wed Aug 8 08:06:13 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Aug 2012 08:06:13 -0500 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: > Dear all, > > For concerning more general geometry of physical domain, for some > situations, > we need to apply the multi-block mesh. The attached figure (or > alternatively the following link) illustrates one of examples. > > https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg > > Have you ever dealt with such kind of case with application of the PETSc > object 'DA' ? > Our intention for this is that you use a DMComposite structure. Let us know if the documentation is unclear on anything, since it is somewhat new. Thanks, Matt > Suppose we build multiple DA(or DM) for governing the multi-block data > structure, > in other words, we respectively build a DA for each block of mesh. > Especially in parallel computing, > how can we manage and control the communications between blocks of data > accurately and efficiently? > And, how can we ensure the correctness of parallel computational results? > > I wonder whether some possible application of PETSc object 'DA' (or DM) is > sufficient for dealing with cases in multi-block mesh. > > > Sincerely regards. > > > Chuei-Ching Chiou > > > -- > Chuei-Ching Chiou, > Ph. D student, > Aerothermal & Plasma Physics Laboratory > Department of Mechanical Engineering > National Chiao Tung University > Hsinchu 30010, TAIWAN > TEL: +886-35712121 ext. 55175 > E-mail: C CChiou2012 at gmail.com > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Wed Aug 8 08:48:09 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Wed, 08 Aug 2012 09:48:09 -0400 Subject: [petsc-users] Field split questions Message-ID: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> Hello all, When using PCFieldSplitSetIS to define splits within splits, should the IS contain indices in the original matrix or in the next highest split level? Also, my matrix is in aij format but one of the sub fields has a block diagonal structure. Is it still possible use block jacobi on this field? Thanks and all the best Colin -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From knepley at gmail.com Wed Aug 8 08:52:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Aug 2012 08:52:12 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> Message-ID: On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: > Hello all, > > When using PCFieldSplitSetIS to define splits within splits, should the IS > contain indices in the original matrix or in the next highest split level? > The next highest split level. You might consider using a DM if you have some structure (like discretization on a mesh). This should handle all the index-wrangling for you. It is new, but intended for just this kind of thing. Also, my matrix is in aij format but one of the sub fields has a block > diagonal structure. Is it still possible use block jacobi on this field? > Yes. Matt > Thanks and all the best > Colin > > > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhalen at gmail.com Wed Aug 8 11:34:50 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Wed, 8 Aug 2012 12:34:50 -0400 Subject: [petsc-users] minor error message compiling Message-ID: I just upgraded to 3.3 and while compiling I get the following error: Traceback (most recent call last): File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in petscdir = os.environ['PETSC_DIR'] File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ raise KeyError(key) KeyError: 'PETSC_DIR' make: [petscnagupgrade] Error 1 (ignored) Seems to occur only when the env variable PETSC_DIR is not defined; the program seems to compile properly inspite of this error. -Nachiket From balay at mcs.anl.gov Wed Aug 8 12:03:40 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 12:03:40 -0500 (CDT) Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > I just upgraded to 3.3 and while compiling I get the following error: > > Traceback (most recent call last): > File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in > petscdir = os.environ['PETSC_DIR'] > File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ > raise KeyError(key) > KeyError: 'PETSC_DIR' > make: [petscnagupgrade] Error 1 (ignored) > > Seems to occur only when the env variable PETSC_DIR is not defined; > the program seems to compile properly inspite of this error. Hm petscnagupgrade.py is primarily invoked by make - and when using PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you get this error. Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] Does the attached patch fix it? patch -Np1 < petscnag.patch Satish -------------- next part -------------- diff --git a/bin/petscnagupgrade.py b/bin/petscnagupgrade.py --- a/bin/petscnagupgrade.py +++ b/bin/petscnagupgrade.py @@ -51,7 +51,12 @@ # # if __name__ == '__main__': - petscdir = os.environ['PETSC_DIR'] + if os.environ.has_key('PETSC_DIR'): + petscdir = os.environ['PETSC_DIR'] + elif os.path.exists(os.path.join('.', 'include', 'petscversion.h')): + petscdir = '.' + else: + sys.exit(0) file = os.path.join(petscdir,'.nagged') if not naggedtoday(file): currentversion(petscdir) From gokhalen at gmail.com Wed Aug 8 12:16:31 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Wed, 8 Aug 2012 13:16:31 -0400 Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 1:03 PM, Satish Balay wrote: > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > >> I just upgraded to 3.3 and while compiling I get the following error: >> >> Traceback (most recent call last): >> File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in >> petscdir = os.environ['PETSC_DIR'] >> File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ >> raise KeyError(key) >> KeyError: 'PETSC_DIR' >> make: [petscnagupgrade] Error 1 (ignored) >> >> Seems to occur only when the env variable PETSC_DIR is not defined; >> the program seems to compile properly inspite of this error. > > Hm petscnagupgrade.py is primarily invoked by make - and when using > PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you > get this error. > > Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] > > Does the attached patch fix it? > > patch -Np1 < petscnag.patch > > Satish Yes your attached patch fixes it. Thanks! I should have been more clear. The problem occured not while compiling PETSc but a small SlepC application. I am including include ${SLEPC_DIR}/conf/slepc_common in my Makefile and relying on it to include PETSC variables. I have a standard Redhat Enterprise Linux system. Thanks! -Nachiket From balay at mcs.anl.gov Wed Aug 8 13:24:43 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 13:24:43 -0500 (CDT) Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > On Wed, Aug 8, 2012 at 1:03 PM, Satish Balay wrote: > > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > > > >> I just upgraded to 3.3 and while compiling I get the following error: > >> > >> Traceback (most recent call last): > >> File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in > >> petscdir = os.environ['PETSC_DIR'] > >> File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ > >> raise KeyError(key) > >> KeyError: 'PETSC_DIR' > >> make: [petscnagupgrade] Error 1 (ignored) > >> > >> Seems to occur only when the env variable PETSC_DIR is not defined; > >> the program seems to compile properly inspite of this error. > > > > Hm petscnagupgrade.py is primarily invoked by make - and when using > > PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you > > get this error. > > > > Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] > > > > Does the attached patch fix it? > > > > patch -Np1 < petscnag.patch > > > > Satish > > Yes your attached patch fixes it. Thanks! > > I should have been more clear. The problem occured not while compiling > PETSc but a small SlepC application. I am including > > include ${SLEPC_DIR}/conf/slepc_common > > in my Makefile and relying on it to include PETSC variables. > > I have a standard Redhat Enterprise Linux system. Ok. Its still not clear to me whats causing this error. Without PETSC_DIR set - how was your make able to get all the way to running '/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py' [because PETSC_DIR was used to locate petscnagupgrade.py at the above path]. Satish From jzhong at scsolutions.com Wed Aug 8 13:37:33 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 18:37:33 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) Message-ID: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> Dear PETSc folks, I have an emerging need to resolve the performance issue in building sparse matrix in PETSc. I am are currently having some performance issues in building the sparse matrix in PETSc. I am trying to input the inverted matrix from ScaLAPACK into PETSc. This matrix is partitioned into series of submatrices with indices identified as block-cyclically distributed pattern. Here are the options I have tested in PETSc to build them into PETSc and their limitations: 1. MatSetValue(s)(...) is very easy to use. However, the assembling process is extremely slow and expensive; 2. MatSetValuesBlocked(..) can speed up the assembly. However, it is not well-fit for block-cyclically distributed matrices; and 3. MatCreateMPIAIJwithArrays(...) seems to be promising. However, it is not straightforward to use for block-cyclically distributed matrices. 4. Customizeing MatSetValuesBlocked(...) such that we can specify the indices for (*mat->ops->setvaluesblocked)(...). However, I have trouble to locate proper line for mat->ops->setvaluesblocked=&unknownFunctionName The question is what other options you have to resolve this performance issue? From your experience, do you guys have a worked example to build PETSc sparse matrix from block-cyclically distributed matrices obtained from ScaLAPACK with good performance? I appreciate your inputs on this issue. Thanks, Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 8 13:44:54 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 13:44:54 -0500 (CDT) Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> Message-ID: First - can you confirm that you've preallocated the matrix correctly? run with '-info' and look for 'mallocs' and make sure its zero. The second cost is the data movement between processes. To speed this up - try a few calls MatAssemblyBegin/End(MAT_FLUSH_ASSEMBLY). i.e: insert insert insert Satish On Wed, 8 Aug 2012, Jinquan Zhong wrote: > Dear PETSc folks, > > I have an emerging need to resolve the performance issue in building sparse matrix in PETSc. > > I am are currently having some performance issues in building the sparse matrix in PETSc. I am trying to input the inverted matrix from ScaLAPACK into PETSc. This matrix is partitioned into series of submatrices with indices identified as block-cyclically distributed pattern. > > Here are the options I have tested in PETSc to build them into PETSc and their limitations: > > > 1. MatSetValue(s)(...) is very easy to use. However, the assembling process is extremely slow and expensive; > > 2. MatSetValuesBlocked(..) can speed up the assembly. However, it is not well-fit for block-cyclically distributed matrices; and > > 3. MatCreateMPIAIJwithArrays(...) seems to be promising. However, it is not straightforward to use for block-cyclically distributed matrices. > > 4. Customizeing MatSetValuesBlocked(...) such that we can specify the indices for (*mat->ops->setvaluesblocked)(...). However, I have trouble to locate proper line for > > mat->ops->setvaluesblocked=&unknownFunctionName > > The question is what other options you have to resolve this performance issue? From your experience, do you guys have a worked example to build PETSc sparse matrix from block-cyclically distributed matrices obtained from ScaLAPACK with good performance? > > I appreciate your inputs on this issue. > > Thanks, > > Jinquan > > From gokhalen at gmail.com Wed Aug 8 13:46:07 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Wed, 8 Aug 2012 14:46:07 -0400 Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 2:24 PM, Satish Balay wrote: > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > >> On Wed, Aug 8, 2012 at 1:03 PM, Satish Balay wrote: >> > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: >> > >> >> I just upgraded to 3.3 and while compiling I get the following error: >> >> >> >> Traceback (most recent call last): >> >> File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in >> >> petscdir = os.environ['PETSC_DIR'] >> >> File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ >> >> raise KeyError(key) >> >> KeyError: 'PETSC_DIR' >> >> make: [petscnagupgrade] Error 1 (ignored) >> >> >> >> Seems to occur only when the env variable PETSC_DIR is not defined; >> >> the program seems to compile properly inspite of this error. >> > >> > Hm petscnagupgrade.py is primarily invoked by make - and when using >> > PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you >> > get this error. >> > >> > Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] >> > >> > Does the attached patch fix it? >> > >> > patch -Np1 < petscnag.patch >> > >> > Satish >> >> Yes your attached patch fixes it. Thanks! >> >> I should have been more clear. The problem occured not while compiling >> PETSc but a small SlepC application. I am including >> >> include ${SLEPC_DIR}/conf/slepc_common >> >> in my Makefile and relying on it to include PETSC variables. >> >> I have a standard Redhat Enterprise Linux system. > > Ok. Its still not clear to me whats causing this error. Without > PETSC_DIR set - how was your make able to get all the way to running > '/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py' [because PETSC_DIR > was used to locate petscnagupgrade.py at the above path]. > > Satish Well, PETSC_DIR was always set in the Makefile (attached), but not in the environment variables in the shell. Setting or unsetting in the bash shell seemed to trigger that error. -Nachiket -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 1833 bytes Desc: not available URL: From jzhong at scsolutions.com Wed Aug 8 13:47:48 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 18:47:48 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> Thanks, Satish. Are you referring to preallocating memory for the size of A before MatSetValues(A, ...)? I didn't do that. Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Wednesday, August 08, 2012 11:45 AM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) First - can you confirm that you've preallocated the matrix correctly? run with '-info' and look for 'mallocs' and make sure its zero. The second cost is the data movement between processes. To speed this up - try a few calls MatAssemblyBegin/End(MAT_FLUSH_ASSEMBLY). i.e: insert insert insert Satish On Wed, 8 Aug 2012, Jinquan Zhong wrote: > Dear PETSc folks, > > I have an emerging need to resolve the performance issue in building sparse matrix in PETSc. > > I am are currently having some performance issues in building the sparse matrix in PETSc. I am trying to input the inverted matrix from ScaLAPACK into PETSc. This matrix is partitioned into series of submatrices with indices identified as block-cyclically distributed pattern. > > Here are the options I have tested in PETSc to build them into PETSc and their limitations: > > > 1. MatSetValue(s)(...) is very easy to use. However, the assembling process is extremely slow and expensive; > > 2. MatSetValuesBlocked(..) can speed up the assembly. However, it is not well-fit for block-cyclically distributed matrices; and > > 3. MatCreateMPIAIJwithArrays(...) seems to be promising. However, it is not straightforward to use for block-cyclically distributed matrices. > > 4. Customizeing MatSetValuesBlocked(...) such that we can specify the indices for (*mat->ops->setvaluesblocked)(...). However, I have trouble to locate proper line for > > mat->ops->setvaluesblocked=&unknownFunctionName > > The question is what other options you have to resolve this performance issue? From your experience, do you guys have a worked example to build PETSc sparse matrix from block-cyclically distributed matrices obtained from ScaLAPACK with good performance? > > I appreciate your inputs on this issue. > > Thanks, > > Jinquan > > From jedbrown at mcs.anl.gov Wed Aug 8 13:54:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 12:54:27 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 12:47 PM, Jinquan Zhong wrote: > Are you referring to preallocating memory for the size of A before > MatSetValues(A, ...)? I didn't do that. This is very important. http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly With petsc-dev, it is best to use MATELEMENTAL for dense matrices. What are you trying to do with this dense martix? -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 8 14:00:08 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 14:00:08 -0500 (CDT) Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > On Wed, Aug 8, 2012 at 2:24 PM, Satish Balay wrote: > > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > > > >> On Wed, Aug 8, 2012 at 1:03 PM, Satish Balay wrote: > >> > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > >> > > >> >> I just upgraded to 3.3 and while compiling I get the following error: > >> >> > >> >> Traceback (most recent call last): > >> >> File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in > >> >> petscdir = os.environ['PETSC_DIR'] > >> >> File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ > >> >> raise KeyError(key) > >> >> KeyError: 'PETSC_DIR' > >> >> make: [petscnagupgrade] Error 1 (ignored) > >> >> > >> >> Seems to occur only when the env variable PETSC_DIR is not defined; > >> >> the program seems to compile properly inspite of this error. > >> > > >> > Hm petscnagupgrade.py is primarily invoked by make - and when using > >> > PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you > >> > get this error. > >> > > >> > Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] > >> > > >> > Does the attached patch fix it? > >> > > >> > patch -Np1 < petscnag.patch > >> > > >> > Satish > >> > >> Yes your attached patch fixes it. Thanks! > >> > >> I should have been more clear. The problem occured not while compiling > >> PETSc but a small SlepC application. I am including > >> > >> include ${SLEPC_DIR}/conf/slepc_common > >> > >> in my Makefile and relying on it to include PETSC variables. > >> > >> I have a standard Redhat Enterprise Linux system. > > > > Ok. Its still not clear to me whats causing this error. Without > > PETSC_DIR set - how was your make able to get all the way to running > > '/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py' [because PETSC_DIR > > was used to locate petscnagupgrade.py at the above path]. > > > > Satish > > > Well, PETSC_DIR was always set in the Makefile (attached), but not in > the environment variables in the shell. Setting or unsetting in the > bash shell seemed to trigger that error. ok. Perhaps you can try using the attached alternate patch. Satish -------------- next part -------------- diff --git a/conf/rules b/conf/rules --- a/conf/rules +++ b/conf/rules @@ -15,7 +15,7 @@ fi petscnagupgrade: - -@${PETSC_DIR}/bin/petscnagupgrade.py + - at PETSC_DIR=${PETSC_DIR} ${PETSC_DIR}/bin/petscnagupgrade.py chk_loc: @if [ ${LOC}foo = foo ] ; then \ From jzhong at scsolutions.com Wed Aug 8 14:00:04 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 19:00:04 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> This makes sense to me. I thought the preallocation only applies to MatCreateMPIAIJwithArrays(?) and other more complicate functions. The matrix coming out from ScaLAPACK will be repopulated into a sparse matrix in a much large scale. Do I need to install elemental package in PETSc in order to use MATELEMENTAL? Thanks, Jed. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 11:54 AM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 12:47 PM, Jinquan Zhong > wrote: Are you referring to preallocating memory for the size of A before MatSetValues(A, ...)? I didn't do that. This is very important. http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly With petsc-dev, it is best to use MATELEMENTAL for dense matrices. What are you trying to do with this dense martix? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhalen at gmail.com Wed Aug 8 14:03:27 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Wed, 8 Aug 2012 15:03:27 -0400 Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 3:00 PM, Satish Balay wrote: > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > >> On Wed, Aug 8, 2012 at 2:24 PM, Satish Balay wrote: >> > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: >> > >> >> On Wed, Aug 8, 2012 at 1:03 PM, Satish Balay wrote: >> >> > On Wed, 8 Aug 2012, Nachiket Gokhale wrote: >> >> > >> >> >> I just upgraded to 3.3 and while compiling I get the following error: >> >> >> >> >> >> Traceback (most recent call last): >> >> >> File "/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py", line 54, in >> >> >> petscdir = os.environ['PETSC_DIR'] >> >> >> File "/usr/local/lib/python2.7/UserDict.py", line 23, in __getitem__ >> >> >> raise KeyError(key) >> >> >> KeyError: 'PETSC_DIR' >> >> >> make: [petscnagupgrade] Error 1 (ignored) >> >> >> >> >> >> Seems to occur only when the env variable PETSC_DIR is not defined; >> >> >> the program seems to compile properly inspite of this error. >> >> > >> >> > Hm petscnagupgrade.py is primarily invoked by make - and when using >> >> > PETSc makefiles - PETSC_DIR is always set - so I'm not sure why you >> >> > get this error. >> >> > >> >> > Are you using PETSc makefiles? What make are you using? [gnu/ bsd/solaris?] >> >> > >> >> > Does the attached patch fix it? >> >> > >> >> > patch -Np1 < petscnag.patch >> >> > >> >> > Satish >> >> >> >> Yes your attached patch fixes it. Thanks! >> >> >> >> I should have been more clear. The problem occured not while compiling >> >> PETSc but a small SlepC application. I am including >> >> >> >> include ${SLEPC_DIR}/conf/slepc_common >> >> >> >> in my Makefile and relying on it to include PETSC variables. >> >> >> >> I have a standard Redhat Enterprise Linux system. >> > >> > Ok. Its still not clear to me whats causing this error. Without >> > PETSC_DIR set - how was your make able to get all the way to running >> > '/opt/petsc/petsc-3.3-p2/bin/petscnagupgrade.py' [because PETSC_DIR >> > was used to locate petscnagupgrade.py at the above path]. >> > >> > Satish >> >> >> Well, PETSC_DIR was always set in the Makefile (attached), but not in >> the environment variables in the shell. Setting or unsetting in the >> bash shell seemed to trigger that error. > > ok. Perhaps you can try using the attached alternate patch. > > Satish This seems to work too. -Nachiket From balay at mcs.anl.gov Wed Aug 8 14:17:54 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 14:17:54 -0500 (CDT) Subject: [petsc-users] minor error message compiling In-Reply-To: References: Message-ID: On Wed, 8 Aug 2012, Nachiket Gokhale wrote: > >> Well, PETSC_DIR was always set in the Makefile (attached), but not in > >> the environment variables in the shell. Setting or unsetting in the > >> bash shell seemed to trigger that error. > > > > ok. Perhaps you can try using the attached alternate patch. > > > > Satish > > This seems to work too. Thanks. Pushed both fixes to petsc-3.3 http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/d0d42c68a021 Satish From jedbrown at mcs.anl.gov Wed Aug 8 14:22:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 13:22:03 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 1:00 PM, Jinquan Zhong wrote: > This makes sense to me. I thought the preallocation only applies to > MatCreateMPIAIJwithArrays(?) and other more complicate functions.**** > > ** ** > > The matrix coming out from ScaLAPACK will be repopulated into a sparse > matrix in a much large scale. > I don't understand this exactly. You are taking the entries of the dense matrix and putting them into a much larger sparse matrix? In that case, you probably never need Elemental, but you definitely need to preallocate. > Do I need to install elemental package in PETSc in order to use > MATELEMENTAL? > Yes, just configure petsc-dev with --download-elemental. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Aug 8 14:32:41 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 19:32:41 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> Thanks, Jed. So would you prefer to using MatCreateAIJ(?) or MatXAIJSetPreallocation(?)? I am having trouble to fill out the parameters. My local dense matrix is A(LDA, LDB). The global size of square matrix is N. What would be the parameters to use them? I used MatCreateAIJ(PETSC_COMM_WORLD, LDA,LDB, N,N, d_nz, d_nnz,o_nz,o_nnz, &A); MatXAIJSetPreallocation(PETSC_COMM_WORLD, 1, d_nz, d_nnz,o_nz,o_nnz, &A); However, I have trouble to figure out the values for d_nz, d_nnz,o_nz, o_nnz since the local A is a block consisting of several sub-blocks from ScaLAPACK. I think MatCreateAIJ and MatXAIJSetPreallocation both take several rows together. Do you think it is still doable to preallocate A using either function? Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 12:22 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 1:00 PM, Jinquan Zhong > wrote: This makes sense to me. I thought the preallocation only applies to MatCreateMPIAIJwithArrays(?) and other more complicate functions. The matrix coming out from ScaLAPACK will be repopulated into a sparse matrix in a much large scale. I don't understand this exactly. You are taking the entries of the dense matrix and putting them into a much larger sparse matrix? In that case, you probably never need Elemental, but you definitely need to preallocate. Do I need to install elemental package in PETSc in order to use MATELEMENTAL? Yes, just configure petsc-dev with --download-elemental. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.null at gmail.com Wed Aug 8 15:10:32 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 8 Aug 2012 21:10:32 +0100 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc Message-ID: Dear all, Assume that I want to convert an indices in application order into petsc order. But I don't know the indices in application is in range. Say I call ao = PETSc.AO().createMapping([4,2,3]) then ao.app2petsc([2,5,3]) will raise an error. is there any way to make return value to [1,-1,2] instead of raising an error? Or there is any way I can handle this error to make it work? Best, Xin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 8 15:11:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 14:11:09 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 1:32 PM, Jinquan Zhong wrote: > Thanks, Jed.**** > > ** ** > > So would you prefer to using MatCreateAIJ(?) or > MatXAIJSetPreallocation(?)? I am having trouble to fill out the > parameters. My local dense matrix is A(LDA, LDB). The global size of > square matrix is N. What would be the parameters to use them? > You said you want to expand this into a larger sparse matrix, therefore you need to figure out how that larger matrix will be organized. I like MatXAIJSetPreallocation() because it works with "blocked" formats, but keep in mind that "blocked" in PETSc does not mean the same as "blocked" for dense matrices. > **** > > ** ** > > I used **** > > MatCreateAIJ(PETSC_COMM_WORLD, LDA,LDB, N,N, d_nz, d_nnz,o_nz,o_nnz, > &A);**** > > MatXAIJSetPreallocation(PETSC_COMM_WORLD, 1, d_nz, d_nnz,o_nz,o_nnz, &A);* > *** > > ** ** > > However, I have trouble to figure out the values for *d_nz, d_nnz,o_nz, > o_nnz* since the local A is a block consisting of several sub-blocks from > ScaLAPACK. I think MatCreateAIJ and MatXAIJSetPreallocation both take > several rows together. > If I understand you correctly, ScaLAPACK blocks don't have anything to do with the sparse matrix structure. > **** > > ** ** > > Do you think it is still doable to preallocate A using either function? > Yes, definitely. Look at examples and/or the users manual for further explanation of preallocation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Aug 8 15:25:25 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 20:25:25 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> Thanks, Jed. I used MatCreateAIJ(PETSC_COMM_WORLD, LDA,LDB, N,N, d_nz, d_nnz,o_nz,o_nnz, &A); MatXAIJSetPreallocation(PETSC_COMM_WORLD, 1, d_nz, d_nnz,o_nz,o_nnz, &A); However, I have trouble to figure out the values for d_nz, d_nnz,o_nz, o_nnz since the local A is a block consisting of several sub-blocks from ScaLAPACK. I think MatCreateAIJ and MatXAIJSetPreallocation both take several rows together. If I understand you correctly, ScaLAPACK blocks don't have anything to do with the sparse matrix structure. ****************************************************************************************************************************************** You are correct. What I meant was how to define the diagonal and off-diagonal parts of each submatrix matrix A (LDA, LDB). For example, in the following matrix, Proc0 Proc1 Proc2 1 2 0 | 0 3 0 | 0 4 Proc0 0 5 6 | 7 0 0 | 8 0 9 0 10 | 11 0 0 | 12 0 ------------------------------------- 13 0 14 | 15 16 17 | 0 0 Proc3 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5 0 0 0 | 22 23 0 | 24 0 ------------------------------------- Proc6 25 26 27 | 0 0 28 | 29 0 30 0 0 | 31 32 33 | 0 34 I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it was based on diagonal and off-diagonal parts. ****************************************************************************************************************************************** Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 8 15:26:26 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 15:26:26 -0500 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc In-Reply-To: References: Message-ID: Hmm, PetscErrorCode AOApplicationToPetsc_Basic(AO ao,PetscInt n,PetscInt *ia) has if (ia[i] >= 0 && ia[i] < N) { ia[i] = aobasic->petsc[ia[i]]; } else { ia[i] = -1; } PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt *ia) has if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, "Invalid input index %D", idex); PetscErrorCode AOApplicationToPetsc_MemoryScalable(AO ao,PetscInt n,PetscInt *ia) has if (ia[i] >= owners[rank] && ia[i] < owners[rank+1] ) { ia[i] = maploc[ia[i]-owners[rank]]; } else { ia[i] = -1 ; /* ia[i] is not in the range of 0 and N-1, maps it to -1 */ } So it seems the intention is to return -1 for out of range not an error. Can the PETSc developer who wrote all this stuff comment? Likely our bug! Also we should fix the manual page for AOApplicationToPetsc() to clearly indicate what happens for out of range values Barry On Aug 8, 2012, at 3:10 PM, Xin Zhao wrote: > Dear all, > > Assume that I want to convert an indices in application order into petsc order. > But I don't know the indices in application is in range. > > Say > I call > ao = PETSc.AO().createMapping([4,2,3]) > > then > ao.app2petsc([2,5,3]) > will raise an error. > > is there any way to make return value to [1,-1,2] instead of raising an error? > > > Or there is any way I can handle this error to make it work? > > Best, > Xin From jedbrown at mcs.anl.gov Wed Aug 8 15:41:56 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 14:41:56 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong wrote: > If I understand you correctly, ScaLAPACK blocks don't have anything to do > with the sparse matrix structure.**** > > ** ** > > > ****************************************************************************************************************************************** > **** > > ** ** > > You are correct. What I meant was how to define the diagonal and > off-diagonal parts of each submatrix matrix A (LDA, LDB). For example, in > the following matrix,**** > > ** ** > > *Proc0* *Proc1* *Proc2***** > > 1 2 0 | 0 3 0 | 0 4**** > > *Proc0* 0 5 6 | 7 0 0 | 8 0**** > > 9 0 10 | 11 0 0 | 12 0**** > > -------------------------------------**** > > 13 0 14 | 15 16 17 | 0 0**** > > *Proc3* 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5**** > > 0 0 0 | 22 23 0 | 24 0**** > > -------------------------------------**** > > *Proc6* 25 26 27 | 0 0 28 | 29 0**** > > 30 0 0 | 31 32 33 | 0 34**** > > ** ** > > I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz > peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it was > based on diagonal and off-diagonal parts. > Throw your 2D block cyclic nonsense out the window. These are sparse matrices and that layout would be terrible. Logically permute your matrices all you want, then define a global ordering and chunk it into contiguous blocks of rows (no partition of columns). Work this out with a pencil and paper. You should have a function that translates row/column pairs from your ordering to our ordering. Now compute the sparsity pattern in the new ordering. (Usually you can figure this out on paper as well.) Then preallocate and call MatSetValues() with the new (row,column) locations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 8 15:45:29 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 15:45:29 -0500 (CDT) Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> Message-ID: On Wed, 8 Aug 2012, Jed Brown wrote: > On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong wrote: > > > If I understand you correctly, ScaLAPACK blocks don't have anything to do > > with the sparse matrix structure.**** > > > > ** ** > > > > > > ****************************************************************************************************************************************** > > **** > > > > ** ** > > > > You are correct. What I meant was how to define the diagonal and > > off-diagonal parts of each submatrix matrix A (LDA, LDB). For example, in > > the following matrix,**** > > > > ** ** > > > > *Proc0* *Proc1* *Proc2***** > > > > 1 2 0 | 0 3 0 | 0 4**** > > > > *Proc0* 0 5 6 | 7 0 0 | 8 0**** > > > > 9 0 10 | 11 0 0 | 12 0**** > > > > -------------------------------------**** > > > > 13 0 14 | 15 16 17 | 0 0**** > > > > *Proc3* 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5**** > > > > 0 0 0 | 22 23 0 | 24 0**** > > > > -------------------------------------**** > > > > *Proc6* 25 26 27 | 0 0 28 | 29 0**** > > > > 30 0 0 | 31 32 33 | 0 34**** > > > > ** ** > > > > I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz > > peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it was > > based on diagonal and off-diagonal parts. > > > > Throw your 2D block cyclic nonsense out the window. These are sparse > matrices and that layout would be terrible. Logically permute your matrices > all you want, then define a global ordering and chunk it into contiguous > blocks of rows (no partition of columns). Work this out with a pencil and > paper. You should have a function that translates row/column pairs from > your ordering to our ordering. Now compute the sparsity pattern in the new > ordering. (Usually you can figure this out on paper as well.) Then > preallocate and call MatSetValues() with the new (row,column) locations. > also check the 'example usage' at: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html Satish From jzhong at scsolutions.com Wed Aug 8 15:53:20 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 20:53:20 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272996@exchange.scsolutions.com> Jed, Your comments got into the core of the application. The dense matrix is predominantly so big ( 800GB) that it would cost too much to move it around in the distributed memory. What I tried to do would be to retain the 800GB already in core from ScaLAPACK untouched and repartition their indices. The next step would be to repartition the indices for each block after I complete the construction of the sparse matrix. This sparse matrix is mainly populated with 800GB entries from dense matrix, along with about 3 GB entries from other matrices. Had it be true as you said, wouldn?t that be a lot of exchange of data among distributed memory? Or I am missing something huge. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 1:42 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong > wrote: If I understand you correctly, ScaLAPACK blocks don't have anything to do with the sparse matrix structure. ****************************************************************************************************************************************** You are correct. What I meant was how to define the diagonal and off-diagonal parts of each submatrix matrix A (LDA, LDB). For example, in the following matrix, Proc0 Proc1 Proc2 1 2 0 | 0 3 0 | 0 4 Proc0 0 5 6 | 7 0 0 | 8 0 9 0 10 | 11 0 0 | 12 0 ------------------------------------- 13 0 14 | 15 16 17 | 0 0 Proc3 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5 0 0 0 | 22 23 0 | 24 0 ------------------------------------- Proc6 25 26 27 | 0 0 28 | 29 0 30 0 0 | 31 32 33 | 0 34 I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it was based on diagonal and off-diagonal parts. Throw your 2D block cyclic nonsense out the window. These are sparse matrices and that layout would be terrible. Logically permute your matrices all you want, then define a global ordering and chunk it into contiguous blocks of rows (no partition of columns). Work this out with a pencil and paper. You should have a function that translates row/column pairs from your ordering to our ordering. Now compute the sparsity pattern in the new ordering. (Usually you can figure this out on paper as well.) Then preallocate and call MatSetValues() with the new (row,column) locations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzj at umich.edu Wed Aug 8 16:01:15 2012 From: lzj at umich.edu (Peter Zhoujie Lu) Date: Wed, 8 Aug 2012 17:01:15 -0400 Subject: [petsc-users] Possible PETSc Linking Problem Message-ID: Hi everyone, I has some issue when I try to compile our own code that uses PETSc. Here is the error message. After some googling, it looks like a linking problem to PETSc. The PETSc compiled and tested without problem. In the makefile, the PETSc linker flags was obtained from PETSC_DIR: include ${PETSC_DIR}/conf/variables PETSC_INCLUDE_FLAGS=${PETSC_CC_INCLUDES} -I$(PETSC_DIR) PETSC_LINKER_FLAGS=${PETSC_LIB} I am using OSX 10.7.4 with gcc(4.8) and PETSc-3.2-p2. The same code was compiled on Linux with both intel and gfortran compilers without problem. I am not 100% sure the issue is due to PETSc... Any suggestions are welcomed. Thanks in advance. Peter Error message: mpif90 SUmb.o SUmb_fortran.o -L../../externals/SU_MPI/lib -L../../externals/ADT/lib -L../../lib -lsumb -lsumb_pyfort -ladt -lsu_mpi -L/usr/X11R6/lib -lX11 -L/usr/local/lib/gcc/x86_64-apple-darwin11.4.0/4.8.0 -L/usr/local/lib -ldl -lmpi_f90 -lmpi_f77 -lgfortran -lgfortran -lquadmath -lm -lm -lmpi_cxx -lstdc++ -lmpi -lopen-rte -lopen-pal -lutil -lSystem -lgcc_ext.10.5 -ldl -L/Users/lvzhoujie/Work/cgns/lib -lcgns -L/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/lib -lpetsc -lmpi -lsuperlu_dist_3.0 -lflapack -lfblas -lmca_common_sm -o ../../bin/SUmb_mac_gfortran_openmpi Undefined symbols for architecture x86_64: "_matcreatempidense_", referenced from: _createextrapetscvars_ in libsumb.a(createExtraPETScVars.o) "_matcreatempiaij_", referenced from: _createstatepetscvars_ in libsumb.a(createStatePETScVars.o) _createspatialpetscvars_ in libsumb.a(createSpatialPETScVars.o) _createcouplingpetscvars_ in libsumb.a(createCouplingPETScVars.o) "_matcreatempibaij_", referenced from: _createstatepetscvars_ in libsumb.a(createStatePETScVars.o) _setupnksolver_ in libsumb.a(setupNKSolver.o) ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 8 16:04:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 15:04:42 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272996@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272996@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 2:53 PM, Jinquan Zhong wrote: > The dense matrix is predominantly so big ( 800GB) that it would cost too > much to move it around in the distributed memory. What I tried to do would > be to retain the 800GB already in core from ScaLAPACK untouched and > repartition their indices. The next step would be to repartition the > indices for each block after I complete the construction of the sparse > matrix. This sparse matrix is mainly populated with 800GB entries from > dense matrix, along with about 3 GB entries from other matrices. **** > > ** ** > > Had it be true as you said, wouldn?t that be a lot of exchange of data > among distributed memory? Or I am missing something huge. > You can change indexing so that Elemental's matrix uses the same memory. PETSc uses yet another indexing. You can look at petsc-dev/src/mat/impls/elemental/matelem.cxx for the index translation between the ordering PETSc presents to the user and the one Elemental uses internally. Anyway, if you configure an Elemental matrix to use your memory, while the (simpler) PETSc ordering has contiguous vectors [1], then you can combine it with traditional sparse matrices using MATNEST. [1] Note that PETSc's ownership ranges really fundamentally refer to the Vec distribution; the matrix entries can be stored elsewhere. There is nothing especially hard about doing the above, but you will have to do some careful indexing math. Look at the MATNEST man page and the Elemental documentation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 8 16:07:03 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 16:07:03 -0500 Subject: [petsc-users] Possible PETSc Linking Problem In-Reply-To: References: Message-ID: <2464E475-0166-4720-8B80-C32F5CA4984A@mcs.anl.gov> The MPI was dropped from the name of each of those "missing" functions; since on one process they are not MPI. See http://www.mcs.anl.gov/petsc/documentation/changes/33.html Barry On Aug 8, 2012, at 4:01 PM, Peter Zhoujie Lu wrote: > Hi everyone, > > I has some issue when I try to compile our own code that uses PETSc. Here is the error message. After some googling, it looks like a linking problem to PETSc. The PETSc compiled and tested without problem. In the makefile, the PETSc linker flags was obtained from PETSC_DIR: > > include ${PETSC_DIR}/conf/variables > PETSC_INCLUDE_FLAGS=${PETSC_CC_INCLUDES} -I$(PETSC_DIR) > PETSC_LINKER_FLAGS=${PETSC_LIB} > > I am using OSX 10.7.4 with gcc(4.8) and PETSc-3.2-p2. The same code was compiled on Linux with both intel and gfortran compilers without problem. > I am not 100% sure the issue is due to PETSc... Any suggestions are welcomed. Thanks in advance. > > Peter > > Error message: > > mpif90 SUmb.o SUmb_fortran.o -L../../externals/SU_MPI/lib -L../../externals/ADT/lib -L../../lib -lsumb -lsumb_pyfort -ladt -lsu_mpi -L/usr/X11R6/lib -lX11 -L/usr/local/lib/gcc/x86_64-apple-darwin11.4.0/4.8.0 -L/usr/local/lib -ldl -lmpi_f90 -lmpi_f77 -lgfortran -lgfortran -lquadmath -lm -lm -lmpi_cxx -lstdc++ -lmpi -lopen-rte -lopen-pal -lutil -lSystem -lgcc_ext.10.5 -ldl -L/Users/lvzhoujie/Work/cgns/lib -lcgns -L/Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/lib -lpetsc -lmpi -lsuperlu_dist_3.0 -lflapack -lfblas -lmca_common_sm -o ../../bin/SUmb_mac_gfortran_openmpi > > Undefined symbols for architecture x86_64: > "_matcreatempidense_", referenced from: > _createextrapetscvars_ in libsumb.a(createExtraPETScVars.o) > "_matcreatempiaij_", referenced from: > _createstatepetscvars_ in libsumb.a(createStatePETScVars.o) > _createspatialpetscvars_ in libsumb.a(createSpatialPETScVars.o) > _createcouplingpetscvars_ in libsumb.a(createCouplingPETScVars.o) > "_matcreatempibaij_", referenced from: > _createstatepetscvars_ in libsumb.a(createStatePETScVars.o) > _setupnksolver_ in libsumb.a(setupNKSolver.o) > ld: symbol(s) not found for architecture x86_64 > collect2: error: ld returned 1 exit status From jzhong at scsolutions.com Wed Aug 8 16:09:50 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 21:09:50 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> Satish, I looked at http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex5s.c.html where MatCreateAIJ(PETSC_COMM_WORLD,rend-rstart,rend-rstart,N,N,5,0,0,0,&J); was used to create J (N,N) with local J dimensioned as (rend-rstart,rend-rstart). For my application, I have LDA=LDB=3 and N=9 in the following ierr = MatCreateAIJ(PETSC_COMM_WORLD,LDA,LDB,N,N,LDA,0,0,0,&A); CHKERRQ(ierr); I got the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Sum of local lengths 27 does not equal global length 9, my local length 3 likely a call to VecSetSizes() or MatSetSizes() is wrong. It does NOT appear PETSc had the 9x9 matrix. It tried to build 27*27 since I am using 9 procs. Is there a way to resolve this local size and global size issue? Thanks, Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Wednesday, August 08, 2012 1:45 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, 8 Aug 2012, Jed Brown wrote: > On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong wrote: > > > If I understand you correctly, ScaLAPACK blocks don't have anything > > to do with the sparse matrix structure.**** > > > > ** ** > > > > > > ******************************************************************** > > ******************************************************************** > > ** > > **** > > > > ** ** > > > > You are correct. What I meant was how to define the diagonal and > > off-diagonal parts of each submatrix matrix A (LDA, LDB). For > > example, in the following matrix,**** > > > > ** ** > > > > *Proc0* *Proc1* *Proc2***** > > > > 1 2 0 | 0 3 0 | 0 4**** > > > > *Proc0* 0 5 6 | 7 0 0 | 8 0**** > > > > 9 0 10 | 11 0 0 | 12 0**** > > > > -------------------------------------**** > > > > 13 0 14 | 15 16 17 | 0 0**** > > > > *Proc3* 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5**** > > > > 0 0 0 | 22 23 0 | 24 0**** > > > > -------------------------------------**** > > > > *Proc6* 25 26 27 | 0 0 28 | 29 0**** > > > > 30 0 0 | 31 32 33 | 0 34**** > > > > ** ** > > > > I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz > > peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it > > was based on diagonal and off-diagonal parts. > > > > Throw your 2D block cyclic nonsense out the window. These are sparse > matrices and that layout would be terrible. Logically permute your > matrices all you want, then define a global ordering and chunk it into > contiguous blocks of rows (no partition of columns). Work this out > with a pencil and paper. You should have a function that translates > row/column pairs from your ordering to our ordering. Now compute the > sparsity pattern in the new ordering. (Usually you can figure this out > on paper as well.) Then preallocate and call MatSetValues() with the new (row,column) locations. > also check the 'example usage' at: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html Satish From jzhong at scsolutions.com Wed Aug 8 16:17:35 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 21:17:35 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272996@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F2729C5@exchange.scsolutions.com> Thanks, Jed. Your solution seems to be comprehensive. I would love to have it in the next phase of implementation. But I am running out of time till this Friday. What I am trying to do is to use a quick fix to improve the performance of building sparse based on the lines we have as follows: ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ?compute Ii,J, and v ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); ?For my application, I have LDA=LDB=3 for local A and N=9 in the following ierr = MatCreateAIJ(PETSC_COMM_WORLD, LDA,LDB, N,N, LDA,0,0,0, &A); CHKERRQ(ierr); I got the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Sum of local lengths 27 does not equal global length 9, my local length 3 likely a call to VecSetSizes() or MatSetSizes() is wrong. It does NOT appear PETSc had the 9x9 matrix. It tried to build 27*27 since I am using 9 procs. Is there a quick way to resolve this local size and global size issue?? Thanks a lot anyway. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 2:05 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 2:53 PM, Jinquan Zhong > wrote: The dense matrix is predominantly so big ( 800GB) that it would cost too much to move it around in the distributed memory. What I tried to do would be to retain the 800GB already in core from ScaLAPACK untouched and repartition their indices. The next step would be to repartition the indices for each block after I complete the construction of the sparse matrix. This sparse matrix is mainly populated with 800GB entries from dense matrix, along with about 3 GB entries from other matrices. Had it be true as you said, wouldn?t that be a lot of exchange of data among distributed memory? Or I am missing something huge. You can change indexing so that Elemental's matrix uses the same memory. PETSc uses yet another indexing. You can look at petsc-dev/src/mat/impls/elemental/matelem.cxx for the index translation between the ordering PETSc presents to the user and the one Elemental uses internally. Anyway, if you configure an Elemental matrix to use your memory, while the (simpler) PETSc ordering has contiguous vectors [1], then you can combine it with traditional sparse matrices using MATNEST. [1] Note that PETSc's ownership ranges really fundamentally refer to the Vec distribution; the matrix entries can be stored elsewhere. There is nothing especially hard about doing the above, but you will have to do some careful indexing math. Look at the MATNEST man page and the Elemental documentation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.null at gmail.com Wed Aug 8 16:18:47 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 8 Aug 2012 22:18:47 +0100 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc In-Reply-To: References: Message-ID: PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt *ia) has if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, "Invalid input index %D", idex); Well.... is it a good idea to rewrite this part as if (low > high) ia[i] = -1; And wait for comments from PETSc developers? Need I recompile petsc4py after compiling PETSc? Best, Xin On Wed, Aug 8, 2012 at 9:26 PM, Barry Smith wrote: > > Hmm, > > PetscErrorCode AOApplicationToPetsc_Basic(AO ao,PetscInt n,PetscInt *ia) > > has > > if (ia[i] >= 0 && ia[i] < N) { > ia[i] = aobasic->petsc[ia[i]]; > } else { > ia[i] = -1; > } > > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt > *ia) > > has > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, > "Invalid input index %D", idex); > > PetscErrorCode AOApplicationToPetsc_MemoryScalable(AO ao,PetscInt > n,PetscInt *ia) > > has > > if (ia[i] >= owners[rank] && ia[i] < owners[rank+1] ) { > ia[i] = maploc[ia[i]-owners[rank]]; > } else { > ia[i] = -1 ; /* ia[i] is not in the range of 0 and N-1, maps it to > -1 */ > } > > So it seems the intention is to return -1 for out of range not an error. > > Can the PETSc developer who wrote all this stuff comment? Likely our > bug! Also we should fix the manual page for AOApplicationToPetsc() to > clearly indicate what happens for out of range values > > Barry > > > On Aug 8, 2012, at 3:10 PM, Xin Zhao wrote: > > > Dear all, > > > > Assume that I want to convert an indices in application order into petsc > order. > > But I don't know the indices in application is in range. > > > > Say > > I call > > ao = PETSc.AO().createMapping([4,2,3]) > > > > then > > ao.app2petsc([2,5,3]) > > will raise an error. > > > > is there any way to make return value to [1,-1,2] instead of raising an > error? > > > > > > Or there is any way I can handle this error to make it work? > > > > Best, > > Xin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.null at gmail.com Wed Aug 8 16:22:26 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 8 Aug 2012 22:22:26 +0100 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc In-Reply-To: References: Message-ID: if (low > high) {ia[i] = -1;continue;} On Wed, Aug 8, 2012 at 10:18 PM, Xin Zhao wrote: > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt > *ia) > > has > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, > "Invalid input index %D", idex); > > Well.... is it a good idea to rewrite this part as > > if (low > high) ia[i] = -1; > > And wait for comments from PETSc developers? > > Need I recompile petsc4py after compiling PETSc? > > Best, > Xin > > On Wed, Aug 8, 2012 at 9:26 PM, Barry Smith wrote: > >> >> Hmm, >> >> PetscErrorCode AOApplicationToPetsc_Basic(AO ao,PetscInt n,PetscInt *ia) >> >> has >> >> if (ia[i] >= 0 && ia[i] < N) { >> ia[i] = aobasic->petsc[ia[i]]; >> } else { >> ia[i] = -1; >> } >> >> PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt >> *ia) >> >> has >> >> if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, >> "Invalid input index %D", idex); >> >> PetscErrorCode AOApplicationToPetsc_MemoryScalable(AO ao,PetscInt >> n,PetscInt *ia) >> >> has >> >> if (ia[i] >= owners[rank] && ia[i] < owners[rank+1] ) { >> ia[i] = maploc[ia[i]-owners[rank]]; >> } else { >> ia[i] = -1 ; /* ia[i] is not in the range of 0 and N-1, maps it >> to -1 */ >> } >> >> So it seems the intention is to return -1 for out of range not an error. >> >> Can the PETSc developer who wrote all this stuff comment? Likely our >> bug! Also we should fix the manual page for AOApplicationToPetsc() to >> clearly indicate what happens for out of range values >> >> Barry >> >> >> On Aug 8, 2012, at 3:10 PM, Xin Zhao wrote: >> >> > Dear all, >> > >> > Assume that I want to convert an indices in application order into >> petsc order. >> > But I don't know the indices in application is in range. >> > >> > Say >> > I call >> > ao = PETSc.AO().createMapping([4,2,3]) >> > >> > then >> > ao.app2petsc([2,5,3]) >> > will raise an error. >> > >> > is there any way to make return value to [1,-1,2] instead of raising an >> error? >> > >> > >> > Or there is any way I can handle this error to make it work? >> > >> > Best, >> > Xin >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 8 16:27:04 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 15:27:04 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> Message-ID: PETSc does NOT use a 2D partition. You need to get that concept out of your head. On Wed, Aug 8, 2012 at 3:09 PM, Jinquan Zhong wrote: > Satish, > > I looked at > > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex5s.c.html > where > > MatCreateAIJ(PETSC_COMM_WORLD,rend-rstart,rend-rstart,N,N,5,0,0,0,&J); > > was used to create J (N,N) with local J dimensioned as > (rend-rstart,rend-rstart). > > For my application, I have LDA=LDB=3 and N=9 in the following > > ierr = MatCreateAIJ(PETSC_COMM_WORLD,LDA,LDB,N,N,LDA,0,0,0,&A); > CHKERRQ(ierr); > > I got the following error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: Sum of local lengths 27 does not equal global length 9, my > local length 3 > likely a call to VecSetSizes() or MatSetSizes() is wrong. > > > It does NOT appear PETSc had the 9x9 matrix. It tried to build 27*27 > since I am using 9 procs. Is there a way to resolve this local size and > global size issue? > > Thanks, > > Jinquan > > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Wednesday, August 08, 2012 1:45 PM > To: PETSc users list > Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) > > On Wed, 8 Aug 2012, Jed Brown wrote: > > > On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong >wrote: > > > > > If I understand you correctly, ScaLAPACK blocks don't have anything > > > to do with the sparse matrix structure.**** > > > > > > ** ** > > > > > > > > > ******************************************************************** > > > ******************************************************************** > > > ** > > > **** > > > > > > ** ** > > > > > > You are correct. What I meant was how to define the diagonal and > > > off-diagonal parts of each submatrix matrix A (LDA, LDB). For > > > example, in the following matrix,**** > > > > > > ** ** > > > > > > *Proc0* *Proc1* *Proc2***** > > > > > > 1 2 0 | 0 3 0 | 0 4**** > > > > > > *Proc0* 0 5 6 | 7 0 0 | 8 0**** > > > > > > 9 0 10 | 11 0 0 | 12 0**** > > > > > > -------------------------------------**** > > > > > > 13 0 14 | 15 16 17 | 0 0**** > > > > > > *Proc3* 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc > 5**** > > > > > > 0 0 0 | 22 23 0 | 24 0**** > > > > > > -------------------------------------**** > > > > > > *Proc6* 25 26 27 | 0 0 28 | 29 0**** > > > > > > 30 0 0 | 31 32 33 | 0 34**** > > > > > > ** ** > > > > > > I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz > > > peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it > > > was based on diagonal and off-diagonal parts. > > > > > > > Throw your 2D block cyclic nonsense out the window. These are sparse > > matrices and that layout would be terrible. Logically permute your > > matrices all you want, then define a global ordering and chunk it into > > contiguous blocks of rows (no partition of columns). Work this out > > with a pencil and paper. You should have a function that translates > > row/column pairs from your ordering to our ordering. Now compute the > > sparsity pattern in the new ordering. (Usually you can figure this out > > on paper as well.) Then preallocate and call MatSetValues() with the new > (row,column) locations. > > > > also check the 'example usage' at: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 8 16:37:26 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 16:37:26 -0500 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc In-Reply-To: References: Message-ID: <4EFCAD10-C8E4-4C59-B943-0B87A6ABCC18@mcs.anl.gov> On Aug 8, 2012, at 4:22 PM, Xin Zhao wrote: > if (low > high) {ia[i] = -1;continue;} > With this change; does it now run as you would like? Barry > On Wed, Aug 8, 2012 at 10:18 PM, Xin Zhao wrote: > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt *ia) > > has > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, "Invalid input index %D", idex); > > Well.... is it a good idea to rewrite this part as > > if (low > high) ia[i] = -1; > > And wait for comments from PETSc developers? > > Need I recompile petsc4py after compiling PETSc? > > Best, > Xin > > On Wed, Aug 8, 2012 at 9:26 PM, Barry Smith wrote: > > Hmm, > > PetscErrorCode AOApplicationToPetsc_Basic(AO ao,PetscInt n,PetscInt *ia) > > has > > if (ia[i] >= 0 && ia[i] < N) { > ia[i] = aobasic->petsc[ia[i]]; > } else { > ia[i] = -1; > } > > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt *ia) > > has > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, "Invalid input index %D", idex); > > PetscErrorCode AOApplicationToPetsc_MemoryScalable(AO ao,PetscInt n,PetscInt *ia) > > has > > if (ia[i] >= owners[rank] && ia[i] < owners[rank+1] ) { > ia[i] = maploc[ia[i]-owners[rank]]; > } else { > ia[i] = -1 ; /* ia[i] is not in the range of 0 and N-1, maps it to -1 */ > } > > So it seems the intention is to return -1 for out of range not an error. > > Can the PETSc developer who wrote all this stuff comment? Likely our bug! Also we should fix the manual page for AOApplicationToPetsc() to clearly indicate what happens for out of range values > > Barry > > > On Aug 8, 2012, at 3:10 PM, Xin Zhao wrote: > > > Dear all, > > > > Assume that I want to convert an indices in application order into petsc order. > > But I don't know the indices in application is in range. > > > > Say > > I call > > ao = PETSc.AO().createMapping([4,2,3]) > > > > then > > ao.app2petsc([2,5,3]) > > will raise an error. > > > > is there any way to make return value to [1,-1,2] instead of raising an error? > > > > > > Or there is any way I can handle this error to make it work? > > > > Best, > > Xin > > > From jzhong at scsolutions.com Wed Aug 8 16:45:54 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 21:45:54 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 2:27 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) PETSc does NOT use a 2D partition. You need to get that concept out of your head. On Wed, Aug 8, 2012 at 3:09 PM, Jinquan Zhong > wrote: Satish, I looked at http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex5s.c.html where MatCreateAIJ(PETSC_COMM_WORLD,rend-rstart,rend-rstart,N,N,5,0,0,0,&J); was used to create J (N,N) with local J dimensioned as (rend-rstart,rend-rstart). For my application, I have LDA=LDB=3 and N=9 in the following ierr = MatCreateAIJ(PETSC_COMM_WORLD,LDA,LDB,N,N,LDA,0,0,0,&A); CHKERRQ(ierr); I got the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Sum of local lengths 27 does not equal global length 9, my local length 3 likely a call to VecSetSizes() or MatSetSizes() is wrong. It does NOT appear PETSc had the 9x9 matrix. It tried to build 27*27 since I am using 9 procs. Is there a way to resolve this local size and global size issue? Thanks, Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Wednesday, August 08, 2012 1:45 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, 8 Aug 2012, Jed Brown wrote: > On Wed, Aug 8, 2012 at 2:25 PM, Jinquan Zhong >wrote: > > > If I understand you correctly, ScaLAPACK blocks don't have anything > > to do with the sparse matrix structure.**** > > > > ** ** > > > > > > ******************************************************************** > > ******************************************************************** > > ** > > **** > > > > ** ** > > > > You are correct. What I meant was how to define the diagonal and > > off-diagonal parts of each submatrix matrix A (LDA, LDB). For > > example, in the following matrix,**** > > > > ** ** > > > > *Proc0* *Proc1* *Proc2***** > > > > 1 2 0 | 0 3 0 | 0 4**** > > > > *Proc0* 0 5 6 | 7 0 0 | 8 0**** > > > > 9 0 10 | 11 0 0 | 12 0**** > > > > -------------------------------------**** > > > > 13 0 14 | 15 16 17 | 0 0**** > > > > *Proc3* 0 18 0 | 19 20 21 | 0 0 <=== owned by Proc 5**** > > > > 0 0 0 | 22 23 0 | 24 0**** > > > > -------------------------------------**** > > > > *Proc6* 25 26 27 | 0 0 28 | 29 0**** > > > > 30 0 0 | 31 32 33 | 0 34**** > > > > ** ** > > > > I am not sure how to fill out the values for d_nz, d_nnz,o_nz, o_nnz > > peoperly for the subblock (0 0 ;0 0; 24 0) owned by Proc 5 since it > > was based on diagonal and off-diagonal parts. > > > > Throw your 2D block cyclic nonsense out the window. These are sparse > matrices and that layout would be terrible. Logically permute your > matrices all you want, then define a global ordering and chunk it into > contiguous blocks of rows (no partition of columns). Work this out > with a pencil and paper. You should have a function that translates > row/column pairs from your ordering to our ordering. Now compute the > sparsity pattern in the new ordering. (Usually you can figure this out > on paper as well.) Then preallocate and call MatSetValues() with the new (row,column) locations. > also check the 'example usage' at: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html Satish -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.null at gmail.com Wed Aug 8 16:52:53 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 8 Aug 2012 22:52:53 +0100 Subject: [petsc-users] return -1 instead of raising an error in ao.app2petsc In-Reply-To: <4EFCAD10-C8E4-4C59-B943-0B87A6ABCC18@mcs.anl.gov> References: <4EFCAD10-C8E4-4C59-B943-0B87A6ABCC18@mcs.anl.gov> Message-ID: I run ao = PETSc.AO().createMapping([5,4,3]) then ao.app2petsc([6,5]) will give array([-1, 0]) I think it works fine. Thanks, Barry. Best, Xin On Wed, Aug 8, 2012 at 10:37 PM, Barry Smith wrote: > > On Aug 8, 2012, at 4:22 PM, Xin Zhao wrote: > > > if (low > high) {ia[i] = -1;continue;} > > > > With this change; does it now run as you would like? > > Barry > > > > On Wed, Aug 8, 2012 at 10:18 PM, Xin Zhao wrote: > > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt > *ia) > > > > has > > > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, > "Invalid input index %D", idex); > > > > Well.... is it a good idea to rewrite this part as > > > > if (low > high) ia[i] = -1; > > > > And wait for comments from PETSc developers? > > > > Need I recompile petsc4py after compiling PETSc? > > > > Best, > > Xin > > > > On Wed, Aug 8, 2012 at 9:26 PM, Barry Smith wrote: > > > > Hmm, > > > > PetscErrorCode AOApplicationToPetsc_Basic(AO ao,PetscInt n,PetscInt *ia) > > > > has > > > > if (ia[i] >= 0 && ia[i] < N) { > > ia[i] = aobasic->petsc[ia[i]]; > > } else { > > ia[i] = -1; > > } > > > > PetscErrorCode AOApplicationToPetsc_Mapping(AO ao, PetscInt n, PetscInt > *ia) > > > > has > > > > if (low > high) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE, > "Invalid input index %D", idex); > > > > PetscErrorCode AOApplicationToPetsc_MemoryScalable(AO ao,PetscInt > n,PetscInt *ia) > > > > has > > > > if (ia[i] >= owners[rank] && ia[i] < owners[rank+1] ) { > > ia[i] = maploc[ia[i]-owners[rank]]; > > } else { > > ia[i] = -1 ; /* ia[i] is not in the range of 0 and N-1, maps it > to -1 */ > > } > > > > So it seems the intention is to return -1 for out of range not an error. > > > > Can the PETSc developer who wrote all this stuff comment? Likely our > bug! Also we should fix the manual page for AOApplicationToPetsc() to > clearly indicate what happens for out of range values > > > > Barry > > > > > > On Aug 8, 2012, at 3:10 PM, Xin Zhao wrote: > > > > > Dear all, > > > > > > Assume that I want to convert an indices in application order into > petsc order. > > > But I don't know the indices in application is in range. > > > > > > Say > > > I call > > > ao = PETSc.AO().createMapping([4,2,3]) > > > > > > then > > > ao.app2petsc([2,5,3]) > > > will raise an error. > > > > > > is there any way to make return value to [1,-1,2] instead of raising > an error? > > > > > > > > > Or there is any way I can handle this error to make it work? > > > > > > Best, > > > Xin > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 8 17:03:32 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 16:03:32 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong wrote: > It is a 1D matrix in PETSc. I just passed it as a 2D matrix in > ScaLAPACK. It is still a 1D matrix when get back into PETSc.**** > > What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Aug 8 17:15:20 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 22:15:20 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> What I am trying to say is that ? allocated 1D array A in PETSc ? passed 1D A into ScaLAPACK ? obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array ? indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK ? constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1. I can have the last step done in using ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); It just took too long. This is the performance issue I am having. To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 3:04 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote: It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Aug 8 17:28:59 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 8 Aug 2012 17:28:59 -0500 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> Message-ID: Jinquan: You need understand some basic concepts in numerical computations, for example: - dense matrix vs. sparse matrix and their data structures - algorithms for solve Ax=b - available software packages, their targeted applications See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references. Hong What I am trying to say is that **** > > **? **allocated 1D array A in PETSc**** > > **? **passed 1D A into ScaLAPACK**** > > **? **obtained the indices of A^-1 computed from ScaLAPACK and > stored them in a 1D array**** > > **? **indexed A^-1 as a 1D array in PETSc using the indices > obtained from ScaLAPACK**** > > **? **constructed a sparse matrix, including A^-1 and others, > using a new set of indices for A^-1. **** > > I can have the last step done in using **** > > ** ** > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > **** > > ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);**** > > ierr = MatSetFromOptions(A);CHKERRQ(ierr);**** > > ierr = MatSetUp(A);CHKERRQ(ierr);**** > > **** > > ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);**** > > ** ** > > It just took too long. This is the performance issue I am having. **** > > ** ** > > To improve the performance, it is necessary for preallocation. The > dilemma is: *MatCreateAIJ* and *MatXCreateAIJ* complained the setting of *d_nz, > d_nnz,o_nz, o_nnz* since there are no diagonal and off-diagonal parts > defined in A^-1.**** > > ** ** > > Jinquan**** > > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Wednesday, August 08, 2012 3:04 PM > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > ** ** > > On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote:**** > > It is a 1D matrix in PETSc. I just passed it as a 2D matrix in > ScaLAPACK. It is still a 1D matrix when get back into PETSc.**** > > ** ** > > What are you trying to say?**** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Aug 8 17:36:34 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 22:36:34 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> Thanks, Prof. Zhang. I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Wednesday, August 08, 2012 3:29 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) Jinquan: You need understand some basic concepts in numerical computations, for example: - dense matrix vs. sparse matrix and their data structures - algorithms for solve Ax=b - available software packages, their targeted applications See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references. Hong What I am trying to say is that * allocated 1D array A in PETSc * passed 1D A into ScaLAPACK * obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array * indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK * constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1. I can have the last step done in using ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); It just took too long. This is the performance issue I am having. To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 3:04 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote: It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Aug 8 17:41:10 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 8 Aug 2012 17:41:10 -0500 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> Message-ID: First question: Are your matrices sparse or dense? How large? What type of math equations, e.g., solve Ax=b or eigenvalue problems? Hong On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong wrote: > Thanks, Prof. Zhang.**** > > ** ** > > I could have used PETSc directly for this application. However, I have to > use this procedure for QA (quality assurance) purposes since this > parallelizing process for a program I am working on is a standard program > widely used in industry. That is, QA purposes have driven us to use > ScaLAPACK for inversion first then PETSc for solution. I wish I could a > big move now to eliminate ScaLAPACK.**** > > ** ** > > Jinquan**** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Hong Zhang > *Sent:* Wednesday, August 08, 2012 3:29 PM > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > ** ** > > Jinquan:**** > > ** ** > > You need understand some basic concepts in numerical computations, for > example:**** > > - dense matrix vs. sparse matrix and their data structures**** > > - algorithms for solve Ax=b**** > > - available software packages, their targeted applications**** > > ** ** > > See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references.**** > > ** ** > > Hong**** > > ** ** > > What I am trying to say is that **** > > ? allocated 1D array A in PETSc**** > > ? passed 1D A into ScaLAPACK**** > > ? obtained the indices of A^-1 computed from ScaLAPACK and stored > them in a 1D array**** > > ? indexed A^-1 as a 1D array in PETSc using the indices obtained > from ScaLAPACK**** > > ? constructed a sparse matrix, including A^-1 and others, using a > new set of indices for A^-1. **** > > I can have the last step done in using **** > > **** > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > **** > > ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);**** > > ierr = MatSetFromOptions(A);CHKERRQ(ierr);**** > > ierr = MatSetUp(A);CHKERRQ(ierr);**** > > **** > > ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);**** > > **** > > It just took too long. This is the performance issue I am having. **** > > **** > > To improve the performance, it is necessary for preallocation. The > dilemma is: *MatCreateAIJ* and *MatXCreateAIJ* complained the setting of *d_nz, > d_nnz,o_nz, o_nnz* since there are no diagonal and off-diagonal parts > defined in A^-1.**** > > **** > > Jinquan**** > > **** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Wednesday, August 08, 2012 3:04 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > **** > > On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote:**** > > It is a 1D matrix in PETSc. I just passed it as a 2D matrix in > ScaLAPACK. It is still a 1D matrix when get back into PETSc.**** > > **** > > What are you trying to say?**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Aug 8 17:45:31 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 8 Aug 2012 22:45:31 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> First question: Are your matrices sparse or dense? * A is a dense matrix at the order of 225,000 of complex*16; How large? * 800GB on disk What type of math equations, e.g., solve Ax=b or eigenvalue problems? * A'x=b, where A'=[A^-1 U B]= sparse matrix at the order of 2 million. B = sparse matrix to be mixed and mingled with A^-1. Hong Jinquan On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong > wrote: Thanks, Prof. Zhang. I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Wednesday, August 08, 2012 3:29 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) Jinquan: You need understand some basic concepts in numerical computations, for example: - dense matrix vs. sparse matrix and their data structures - algorithms for solve Ax=b - available software packages, their targeted applications See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references. Hong What I am trying to say is that * allocated 1D array A in PETSc * passed 1D A into ScaLAPACK * obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array * indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK * constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1. I can have the last step done in using ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); It just took too long. This is the performance issue I am having. To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 3:04 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote: It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Wed Aug 8 19:54:55 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Wed, 08 Aug 2012 20:54:55 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> Message-ID: <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc version 3.1p8 callable from fortran? I am getting the following error when I compile the fortran code, and I cant figure out if this is an error in my code or what. Undefined symbols for architecture x86_64: "_pcfieldsplitgetsubksp_", referenced from: _umacr6_ in umacr6.o ld: symbol(s) not found for architecture x86_64 thanks Colin Quoting Matthew Knepley : > On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: > >> Hello all, >> >> When using PCFieldSplitSetIS to define splits within splits, should the IS >> contain indices in the original matrix or in the next highest split level? >> > > The next highest split level. You might consider using a DM if you have > some structure (like discretization on a mesh). > This should handle all the index-wrangling for you. It is new, but intended > for just this kind of thing. > > Also, my matrix is in aij format but one of the sub fields has a block >> diagonal structure. Is it still possible use block jacobi on this field? >> > > Yes. > > Matt > > >> Thanks and all the best >> Colin >> >> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From hzhang at mcs.anl.gov Wed Aug 8 20:02:21 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 8 Aug 2012 20:02:21 -0500 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> Message-ID: Jinquan: Has this problem or similar problem been solved using scalapack? Your problem is virtually a dense problem, why choose petsc library? What type of computer, number of cores, to be used for this problem? Hong First question:**** > > Are your matrices sparse or dense?**** > > **? **A is a dense matrix at the order of 225,000 of complex*16; > **** > > ** ** > > How large? **** > > **? **800GB on disk**** > > ** ** > > What type of math equations, e.g., solve Ax=b or eigenvalue problems?**** > > **? **A?x=b, where A?=[A^-1 U B]= sparse matrix at the order of 2 > million. B = sparse matrix to be mixed and mingled with A^-1.**** > > ** ** > > Hong**** > > ** ** > > Jinquan**** > > On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong > wrote:**** > > Thanks, Prof. Zhang.**** > > **** > > I could have used PETSc directly for this application. However, I have to > use this procedure for QA (quality assurance) purposes since this > parallelizing process for a program I am working on is a standard program > widely used in industry. That is, QA purposes have driven us to use > ScaLAPACK for inversion first then PETSc for solution. I wish I could a > big move now to eliminate ScaLAPACK.**** > > **** > > Jinquan**** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Hong Zhang > *Sent:* Wednesday, August 08, 2012 3:29 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > **** > > Jinquan:**** > > **** > > You need understand some basic concepts in numerical computations, for > example:**** > > - dense matrix vs. sparse matrix and their data structures**** > > - algorithms for solve Ax=b**** > > - available software packages, their targeted applications**** > > **** > > See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references.**** > > **** > > Hong**** > > **** > > What I am trying to say is that **** > > ? allocated 1D array A in PETSc**** > > ? passed 1D A into ScaLAPACK**** > > ? obtained the indices of A^-1 computed from ScaLAPACK and stored > them in a 1D array**** > > ? indexed A^-1 as a 1D array in PETSc using the indices obtained > from ScaLAPACK**** > > ? constructed a sparse matrix, including A^-1 and others, using a > new set of indices for A^-1. **** > > I can have the last step done in using **** > > **** > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > **** > > ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);**** > > ierr = MatSetFromOptions(A);CHKERRQ(ierr);**** > > ierr = MatSetUp(A);CHKERRQ(ierr);**** > > **** > > ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);**** > > **** > > It just took too long. This is the performance issue I am having. **** > > **** > > To improve the performance, it is necessary for preallocation. The > dilemma is: *MatCreateAIJ* and *MatXCreateAIJ* complained the setting of *d_nz, > d_nnz,o_nz, o_nnz* since there are no diagonal and off-diagonal parts > defined in A^-1.**** > > **** > > Jinquan**** > > **** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Wednesday, August 08, 2012 3:04 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Customizeing MatSetValuesBlocked(...)**** > > **** > > On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote:**** > > It is a 1D matrix in PETSc. I just passed it as a 2D matrix in > ScaLAPACK. It is still a 1D matrix when get back into PETSc.**** > > **** > > What are you trying to say?**** > > **** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 8 20:12:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 8 Aug 2012 19:12:43 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> Message-ID: On Wed, Aug 8, 2012 at 4:45 PM, Jinquan Zhong wrote: > What type of math equations, e.g., solve Ax=b or eigenvalue problems?**** > > **? **A?x=b, where A?=[A^-1 U B]= > What does the notation above mean? > sparse matrix at the order of 2 million. B = sparse matrix to be mixed > and mingled with A^-1. > What are the dimensions and sparsity of each piece. If you have an "inverse" floating around, what sort of operator is it the inverse of. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 8 20:28:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 20:28:07 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> Message-ID: <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> Colin, I'm sorry we never had a FORTRAN interface for this routine. You'll need to switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html to use that function. Barry On Aug 8, 2012, at 7:54 PM, Colin McAuliffe wrote: > Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc version 3.1p8 callable from fortran? I am getting the following error when I compile the fortran code, and I cant figure out if this is an error in my code or what. > > Undefined symbols for architecture x86_64: > "_pcfieldsplitgetsubksp_", referenced from: > _umacr6_ in umacr6.o > ld: symbol(s) not found for architecture x86_64 > > > thanks > Colin > > > Quoting Matthew Knepley : > >> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: >> >>> Hello all, >>> >>> When using PCFieldSplitSetIS to define splits within splits, should the IS >>> contain indices in the original matrix or in the next highest split level? >>> >> >> The next highest split level. You might consider using a DM if you have >> some structure (like discretization on a mesh). >> This should handle all the index-wrangling for you. It is new, but intended >> for just this kind of thing. >> >> Also, my matrix is in aij format but one of the sub fields has a block >>> diagonal structure. Is it still possible use block jacobi on this field? >>> >> >> Yes. >> >> Matt >> >> >>> Thanks and all the best >>> Colin >>> >>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics From cjm2176 at columbia.edu Wed Aug 8 20:52:57 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Wed, 08 Aug 2012 21:52:57 -0400 Subject: [petsc-users] Field split questions In-Reply-To: <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> Message-ID: <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> Is there an alternative to this routine that will allow me to define hierarchical field splits in 3.1p8? It will be a fairly significant undertaking to upgrade the fortran code I am using to be compatible with newer versions of petsc. Colin Quoting Barry Smith : > > Colin, > > I'm sorry we never had a FORTRAN interface for this routine. > You'll need to switch to petsc-dev > http://www.mcs.anl.gov/petsc/developers/index.html to use that > function. > > Barry > > > On Aug 8, 2012, at 7:54 PM, Colin McAuliffe wrote: > >> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc >> version 3.1p8 callable from fortran? I am getting the following >> error when I compile the fortran code, and I cant figure out if >> this is an error in my code or what. >> >> Undefined symbols for architecture x86_64: >> "_pcfieldsplitgetsubksp_", referenced from: >> _umacr6_ in umacr6.o >> ld: symbol(s) not found for architecture x86_64 >> >> >> thanks >> Colin >> >> >> Quoting Matthew Knepley : >> >>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe >>> wrote: >>> >>>> Hello all, >>>> >>>> When using PCFieldSplitSetIS to define splits within splits, should the IS >>>> contain indices in the original matrix or in the next highest split level? >>>> >>> >>> The next highest split level. You might consider using a DM if you have >>> some structure (like discretization on a mesh). >>> This should handle all the index-wrangling for you. It is new, but intended >>> for just this kind of thing. >>> >>> Also, my matrix is in aij format but one of the sub fields has a block >>>> diagonal structure. Is it still possible use block jacobi on this field? >>>> >>> >>> Yes. >>> >>> Matt >>> >>> >>>> Thanks and all the best >>>> Colin >>>> >>>> >>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics > > > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From bsmith at mcs.anl.gov Wed Aug 8 21:02:23 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 21:02:23 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> Message-ID: <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> Well the field split stuff in 3.1 is rather iffy anyways. You really need to take the couple hours and do the upgrade before trying to add new features to the fortran code you are using. The time to upgrade will be much less than time spent in work arounds. Barry On Aug 8, 2012, at 8:52 PM, Colin McAuliffe wrote: > Is there an alternative to this routine that will allow me to define hierarchical field splits in 3.1p8? It will be a fairly significant undertaking to upgrade the fortran code I am using to be compatible with newer versions of petsc. > > Colin > > Quoting Barry Smith : > >> >> Colin, >> >> I'm sorry we never had a FORTRAN interface for this routine. You'll need to switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html to use that function. >> >> Barry >> >> >> On Aug 8, 2012, at 7:54 PM, Colin McAuliffe wrote: >> >>> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc version 3.1p8 callable from fortran? I am getting the following error when I compile the fortran code, and I cant figure out if this is an error in my code or what. >>> >>> Undefined symbols for architecture x86_64: >>> "_pcfieldsplitgetsubksp_", referenced from: >>> _umacr6_ in umacr6.o >>> ld: symbol(s) not found for architecture x86_64 >>> >>> >>> thanks >>> Colin >>> >>> >>> Quoting Matthew Knepley : >>> >>>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: >>>> >>>>> Hello all, >>>>> >>>>> When using PCFieldSplitSetIS to define splits within splits, should the IS >>>>> contain indices in the original matrix or in the next highest split level? >>>>> >>>> >>>> The next highest split level. You might consider using a DM if you have >>>> some structure (like discretization on a mesh). >>>> This should handle all the index-wrangling for you. It is new, but intended >>>> for just this kind of thing. >>>> >>>> Also, my matrix is in aij format but one of the sub fields has a block >>>>> diagonal structure. Is it still possible use block jacobi on this field? >>>>> >>>> >>>> Yes. >>>> >>>> Matt >>>> >>>> >>>>> Thanks and all the best >>>>> Colin >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >> >> >> > > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics From balay at mcs.anl.gov Wed Aug 8 21:08:13 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 8 Aug 2012 21:08:13 -0500 (CDT) Subject: [petsc-users] Field split questions In-Reply-To: <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> Message-ID: Barry - you forgot to commit/push the interface files in src/ksp/pc/impls/fieldsplit/ftn-custom. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/40f9515d8b01 Perhaps this interface addition can go into petsc-3.3? That patch [with a minor fix] can be easily applied to petsc-3.1 to get the link working. [old iffy functionality is a different matter :)] Satish On Wed, 8 Aug 2012, Barry Smith wrote: > > Well the field split stuff in 3.1 is rather iffy anyways. You really need to take the couple hours and do the upgrade before trying to add new features to the fortran code you are using. The time to upgrade will be much less than time spent in work arounds. > > Barry > > On Aug 8, 2012, at 8:52 PM, Colin McAuliffe wrote: > > > Is there an alternative to this routine that will allow me to define hierarchical field splits in 3.1p8? It will be a fairly significant undertaking to upgrade the fortran code I am using to be compatible with newer versions of petsc. > > > > Colin > > > > Quoting Barry Smith : > > > >> > >> Colin, > >> > >> I'm sorry we never had a FORTRAN interface for this routine. You'll need to switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html to use that function. > >> > >> Barry > >> > >> > >> On Aug 8, 2012, at 7:54 PM, Colin McAuliffe wrote: > >> > >>> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc version 3.1p8 callable from fortran? I am getting the following error when I compile the fortran code, and I cant figure out if this is an error in my code or what. > >>> > >>> Undefined symbols for architecture x86_64: > >>> "_pcfieldsplitgetsubksp_", referenced from: > >>> _umacr6_ in umacr6.o > >>> ld: symbol(s) not found for architecture x86_64 > >>> > >>> > >>> thanks > >>> Colin > >>> > >>> > >>> Quoting Matthew Knepley : > >>> > >>>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: > >>>> > >>>>> Hello all, > >>>>> > >>>>> When using PCFieldSplitSetIS to define splits within splits, should the IS > >>>>> contain indices in the original matrix or in the next highest split level? > >>>>> > >>>> > >>>> The next highest split level. You might consider using a DM if you have > >>>> some structure (like discretization on a mesh). > >>>> This should handle all the index-wrangling for you. It is new, but intended > >>>> for just this kind of thing. > >>>> > >>>> Also, my matrix is in aij format but one of the sub fields has a block > >>>>> diagonal structure. Is it still possible use block jacobi on this field? > >>>>> > >>>> > >>>> Yes. > >>>> > >>>> Matt > >>>> > >>>> > >>>>> Thanks and all the best > >>>>> Colin > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Colin McAuliffe > >>>>> PhD Candidate > >>>>> Columbia University > >>>>> Department of Civil Engineering and Engineering Mechanics > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> What most experimenters take for granted before they begin their > >>>> experiments is infinitely more interesting than any results to which their > >>>> experiments lead. > >>>> -- Norbert Wiener > >>>> > >>> > >>> > >>> > >>> -- > >>> Colin McAuliffe > >>> PhD Candidate > >>> Columbia University > >>> Department of Civil Engineering and Engineering Mechanics > >> > >> > >> > > > > > > > > -- > > Colin McAuliffe > > PhD Candidate > > Columbia University > > Department of Civil Engineering and Engineering Mechanics > > From bsmith at mcs.anl.gov Wed Aug 8 21:12:01 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Aug 2012 21:12:01 -0500 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> Message-ID: <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> On Aug 8, 2012, at 9:08 PM, Satish Balay wrote: > Barry - you forgot to commit/push the interface files in > src/ksp/pc/impls/fieldsplit/ftn-custom. Thanks. I forgot to hg add the new files :-( > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/40f9515d8b01 > > Perhaps this interface addition can go into petsc-3.3? No > > That patch [with a minor fix] can be easily applied to petsc-3.1 to > get the link working. [old iffy functionality is a different matter :)] Users are free to shot themselves in the foot if they want to so Colin certainly could back-port this but I recommend upgrading. Barry > > Satish > > On Wed, 8 Aug 2012, Barry Smith wrote: > >> >> Well the field split stuff in 3.1 is rather iffy anyways. You really need to take the couple hours and do the upgrade before trying to add new features to the fortran code you are using. The time to upgrade will be much less than time spent in work arounds. >> >> Barry >> >> On Aug 8, 2012, at 8:52 PM, Colin McAuliffe wrote: >> >>> Is there an alternative to this routine that will allow me to define hierarchical field splits in 3.1p8? It will be a fairly significant undertaking to upgrade the fortran code I am using to be compatible with newer versions of petsc. >>> >>> Colin >>> >>> Quoting Barry Smith : >>> >>>> >>>> Colin, >>>> >>>> I'm sorry we never had a FORTRAN interface for this routine. You'll need to switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html to use that function. >>>> >>>> Barry >>>> >>>> >>>> On Aug 8, 2012, at 7:54 PM, Colin McAuliffe wrote: >>>> >>>>> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc version 3.1p8 callable from fortran? I am getting the following error when I compile the fortran code, and I cant figure out if this is an error in my code or what. >>>>> >>>>> Undefined symbols for architecture x86_64: >>>>> "_pcfieldsplitgetsubksp_", referenced from: >>>>> _umacr6_ in umacr6.o >>>>> ld: symbol(s) not found for architecture x86_64 >>>>> >>>>> >>>>> thanks >>>>> Colin >>>>> >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe wrote: >>>>>> >>>>>>> Hello all, >>>>>>> >>>>>>> When using PCFieldSplitSetIS to define splits within splits, should the IS >>>>>>> contain indices in the original matrix or in the next highest split level? >>>>>>> >>>>>> >>>>>> The next highest split level. You might consider using a DM if you have >>>>>> some structure (like discretization on a mesh). >>>>>> This should handle all the index-wrangling for you. It is new, but intended >>>>>> for just this kind of thing. >>>>>> >>>>>> Also, my matrix is in aij format but one of the sub fields has a block >>>>>>> diagonal structure. Is it still possible use block jacobi on this field? >>>>>>> >>>>>> >>>>>> Yes. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks and all the best >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >> >> > From knepley at gmail.com Wed Aug 8 21:20:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Aug 2012 21:20:55 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> Message-ID: On Wed, Aug 8, 2012 at 9:12 PM, Barry Smith wrote: > > On Aug 8, 2012, at 9:08 PM, Satish Balay wrote: > > > Barry - you forgot to commit/push the interface files in > > src/ksp/pc/impls/fieldsplit/ftn-custom. > > Thanks. I forgot to hg add the new files :-( > > > > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/40f9515d8b01 > > > > Perhaps this interface addition can go into petsc-3.3? > > No > > > > > That patch [with a minor fix] can be easily applied to petsc-3.1 to > > get the link working. [old iffy functionality is a different matter :)] > > Users are free to shot themselves in the foot if they want to so Colin > certainly could back-port this but I recommend upgrading. Colin, I also recommend upgrading since with the new stuff, you should be able to do everything you want to do from the command line without messing around pulling out KSP objects. Matt > > Barry > > > > > Satish > > > > On Wed, 8 Aug 2012, Barry Smith wrote: > > > >> > >> Well the field split stuff in 3.1 is rather iffy anyways. You really > need to take the couple hours and do the upgrade before trying to add new > features to the fortran code you are using. The time to upgrade will be > much less than time spent in work arounds. > >> > >> Barry > >> > >> On Aug 8, 2012, at 8:52 PM, Colin McAuliffe > wrote: > >> > >>> Is there an alternative to this routine that will allow me to define > hierarchical field splits in 3.1p8? It will be a fairly significant > undertaking to upgrade the fortran code I am using to be compatible with > newer versions of petsc. > >>> > >>> Colin > >>> > >>> Quoting Barry Smith : > >>> > >>>> > >>>> Colin, > >>>> > >>>> I'm sorry we never had a FORTRAN interface for this routine. > You'll need to switch to petsc-dev > http://www.mcs.anl.gov/petsc/developers/index.html to use that function. > >>>> > >>>> Barry > >>>> > >>>> > >>>> On Aug 8, 2012, at 7:54 PM, Colin McAuliffe > wrote: > >>>> > >>>>> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc > version 3.1p8 callable from fortran? I am getting the following error > when I compile the fortran code, and I cant figure out if this is an error > in my code or what. > >>>>> > >>>>> Undefined symbols for architecture x86_64: > >>>>> "_pcfieldsplitgetsubksp_", referenced from: > >>>>> _umacr6_ in umacr6.o > >>>>> ld: symbol(s) not found for architecture x86_64 > >>>>> > >>>>> > >>>>> thanks > >>>>> Colin > >>>>> > >>>>> > >>>>> Quoting Matthew Knepley : > >>>>> > >>>>>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe < > cjm2176 at columbia.edu>wrote: > >>>>>> > >>>>>>> Hello all, > >>>>>>> > >>>>>>> When using PCFieldSplitSetIS to define splits within splits, > should the IS > >>>>>>> contain indices in the original matrix or in the next highest > split level? > >>>>>>> > >>>>>> > >>>>>> The next highest split level. You might consider using a DM if you > have > >>>>>> some structure (like discretization on a mesh). > >>>>>> This should handle all the index-wrangling for you. It is new, but > intended > >>>>>> for just this kind of thing. > >>>>>> > >>>>>> Also, my matrix is in aij format but one of the sub fields has a > block > >>>>>>> diagonal structure. Is it still possible use block jacobi on this > field? > >>>>>>> > >>>>>> > >>>>>> Yes. > >>>>>> > >>>>>> Matt > >>>>>> > >>>>>> > >>>>>>> Thanks and all the best > >>>>>>> Colin > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Colin McAuliffe > >>>>>>> PhD Candidate > >>>>>>> Columbia University > >>>>>>> Department of Civil Engineering and Engineering Mechanics > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> What most experimenters take for granted before they begin their > >>>>>> experiments is infinitely more interesting than any results to > which their > >>>>>> experiments lead. > >>>>>> -- Norbert Wiener > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Colin McAuliffe > >>>>> PhD Candidate > >>>>> Columbia University > >>>>> Department of Civil Engineering and Engineering Mechanics > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Colin McAuliffe > >>> PhD Candidate > >>> Columbia University > >>> Department of Civil Engineering and Engineering Mechanics > >> > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Wed Aug 8 22:23:50 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Wed, 08 Aug 2012 23:23:50 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> Message-ID: <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> Thanks all, regarding use of DM in 3.3, is the procedure now to create the fields with PCFieldSplitSetIS and then use DMCreateFieldDecompositionDM to create a new DM based from the new fields and the DM for the original problem? Colin Quoting Matthew Knepley : > On Wed, Aug 8, 2012 at 9:12 PM, Barry Smith wrote: > >> >> On Aug 8, 2012, at 9:08 PM, Satish Balay wrote: >> >> > Barry - you forgot to commit/push the interface files in >> > src/ksp/pc/impls/fieldsplit/ftn-custom. >> >> Thanks. I forgot to hg add the new files :-( >> >> > >> > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/40f9515d8b01 >> > >> > Perhaps this interface addition can go into petsc-3.3? >> >> No >> >> > >> > That patch [with a minor fix] can be easily applied to petsc-3.1 to >> > get the link working. [old iffy functionality is a different matter :)] >> >> Users are free to shot themselves in the foot if they want to so Colin >> certainly could back-port this but I recommend upgrading. > > > Colin, I also recommend upgrading since with the new stuff, you should be > able to do everything you want > to do from the command line without messing around pulling out KSP objects. > > Matt > > >> >> Barry >> >> > >> > Satish >> > >> > On Wed, 8 Aug 2012, Barry Smith wrote: >> > >> >> >> >> Well the field split stuff in 3.1 is rather iffy anyways. You really >> need to take the couple hours and do the upgrade before trying to add new >> features to the fortran code you are using. The time to upgrade will be >> much less than time spent in work arounds. >> >> >> >> Barry >> >> >> >> On Aug 8, 2012, at 8:52 PM, Colin McAuliffe >> wrote: >> >> >> >>> Is there an alternative to this routine that will allow me to define >> hierarchical field splits in 3.1p8? It will be a fairly significant >> undertaking to upgrade the fortran code I am using to be compatible with >> newer versions of petsc. >> >>> >> >>> Colin >> >>> >> >>> Quoting Barry Smith : >> >>> >> >>>> >> >>>> Colin, >> >>>> >> >>>> I'm sorry we never had a FORTRAN interface for this routine. >> You'll need to switch to petsc-dev >> http://www.mcs.anl.gov/petsc/developers/index.html to use that function. >> >>>> >> >>>> Barry >> >>>> >> >>>> >> >>>> On Aug 8, 2012, at 7:54 PM, Colin McAuliffe >> wrote: >> >>>> >> >>>>> Thanks I will look into DM. Is PCFieldSplitGetSubKSP in petsc >> version 3.1p8 callable from fortran? I am getting the following error >> when I compile the fortran code, and I cant figure out if this is an error >> in my code or what. >> >>>>> >> >>>>> Undefined symbols for architecture x86_64: >> >>>>> "_pcfieldsplitgetsubksp_", referenced from: >> >>>>> _umacr6_ in umacr6.o >> >>>>> ld: symbol(s) not found for architecture x86_64 >> >>>>> >> >>>>> >> >>>>> thanks >> >>>>> Colin >> >>>>> >> >>>>> >> >>>>> Quoting Matthew Knepley : >> >>>>> >> >>>>>> On Wed, Aug 8, 2012 at 8:48 AM, Colin McAuliffe < >> cjm2176 at columbia.edu>wrote: >> >>>>>> >> >>>>>>> Hello all, >> >>>>>>> >> >>>>>>> When using PCFieldSplitSetIS to define splits within splits, >> should the IS >> >>>>>>> contain indices in the original matrix or in the next highest >> split level? >> >>>>>>> >> >>>>>> >> >>>>>> The next highest split level. You might consider using a DM if you >> have >> >>>>>> some structure (like discretization on a mesh). >> >>>>>> This should handle all the index-wrangling for you. It is new, but >> intended >> >>>>>> for just this kind of thing. >> >>>>>> >> >>>>>> Also, my matrix is in aij format but one of the sub fields has a >> block >> >>>>>>> diagonal structure. Is it still possible use block jacobi on this >> field? >> >>>>>>> >> >>>>>> >> >>>>>> Yes. >> >>>>>> >> >>>>>> Matt >> >>>>>> >> >>>>>> >> >>>>>>> Thanks and all the best >> >>>>>>> Colin >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Colin McAuliffe >> >>>>>>> PhD Candidate >> >>>>>>> Columbia University >> >>>>>>> Department of Civil Engineering and Engineering Mechanics >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> What most experimenters take for granted before they begin their >> >>>>>> experiments is infinitely more interesting than any results to >> which their >> >>>>>> experiments lead. >> >>>>>> -- Norbert Wiener >> >>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Colin McAuliffe >> >>>>> PhD Candidate >> >>>>> Columbia University >> >>>>> Department of Civil Engineering and Engineering Mechanics >> >>>> >> >>>> >> >>>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Colin McAuliffe >> >>> PhD Candidate >> >>> Columbia University >> >>> Department of Civil Engineering and Engineering Mechanics >> >> >> >> >> > >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From knepley at gmail.com Wed Aug 8 22:51:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Aug 2012 22:51:01 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> Message-ID: On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe wrote: > Thanks all, regarding use of DM in 3.3, is the procedure now to create the > fields with PCFieldSplitSetIS and then use DMCreateFieldDecompositionDM to > create a new DM based from the new fields and the DM for the original > problem? > 1) Use petsc-dev 2) PCFieldSplitSetIS() is independent. This allows you to define splits however you want, but then recursive gets harder 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields apart at once 4) In petsc-dev, it uses DMCreateSubDM() which can split off any combination of fields, which from the command line is something like -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 Matt > Colin > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 8 22:52:03 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Aug 2012 22:52:03 -0500 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> Message-ID: On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley wrote: > On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe wrote: > >> Thanks all, regarding use of DM in 3.3, is the procedure now to create >> the fields with PCFieldSplitSetIS and then use DMCreateFieldDecompositionDM >> to create a new DM based from the new fields and the DM for the original >> problem? >> > > 1) Use petsc-dev > > 2) PCFieldSplitSetIS() is independent. This allows you to define splits > however you want, but then recursive gets harder > > 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields > apart at once > > 4) In petsc-dev, it uses DMCreateSubDM() which can split off any > combination of fields, which from the command line is something like > > -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 > I should have shown recursive: -fieldsplit_0_pc_type fieldsplit will split 2,0 into two blocks. Matt > Matt > > >> Colin >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.hisch at gmail.com Thu Aug 9 04:06:50 2012 From: t.hisch at gmail.com (Thomas Hisch) Date: Thu, 9 Aug 2012 11:06:50 +0200 Subject: [petsc-users] Speedup calculation of a high number of generalized eigenvalue problems In-Reply-To: <18E86B24-AB21-472C-B8F1-6577D0D3FC69@dsic.upv.es> References: <18E86B24-AB21-472C-B8F1-6577D0D3FC69@dsic.upv.es> Message-ID: On Wed, Aug 8, 2012 at 1:43 PM, Jose E. Roman wrote: > > El 08/08/2012, a las 11:52, Thomas Hisch escribi?: > >> Hello list, >> >> What can be done to speedup the calculation of the eigenvals and >> eigenvecs of many (~1000)!! generalized EVP, where the operator A is >> constant for all GEVPs and only B is different (but the nonzero >> pattern, size, type,.. are the same for all B). For now I create an >> EPS object for every pair (A,B) like in the following code: (Note: B >> is updated in other functions) >> >> >> void SlepcEigenSolver::Solve(ngbla::Matrix &evecs, >> ngbla::Vector &evals) >> { >> .... >> >> Vec xev; >> MatGetVecs(A_,PETSC_NULL,&xev); >> PetscPrintf(PETSC_COMM_WORLD,"Create EPS object and solve eigensystem\n"); >> EPSCreate(PETSC_COMM_WORLD, &eps_); >> EPSSetOperators(eps_,A_,B_); >> EPSSetDimensions(eps_, GetNev(), PETSC_DECIDE, PETSC_DECIDE); >> EPSSetFromOptions(eps_); >> >> PetscScalar target = GetShift(); >> target *= target; >> >> EPSSetTarget(eps_,target); >> EPSSetWhichEigenpairs(eps_,EPS_TARGET_MAGNITUDE); >> >> ST myst; >> EPSGetST(eps_,&myst); >> STSetType(myst,STSINVERT); >> >> PetscLogDouble t1,t2,t3; >> PetscGetTime(&t1); >> EPSSetUp(eps_); >> PetscGetTime(&t2); >> EPSSolve(eps_); >> PetscGetTime(&t3); >> PetscPrintf(PETSC_COMM_WORLD," -> EPS Solving Done\n\n"); >> PetscPrintf(PETSC_COMM_WORLD,"EPS: Elapsed time: %G (setup), %G >> (solve)\n",t2-t1,t3-t2); >> >> PetscInt its; >> EPSGetIterationNumber(eps_,&its); >> >> PetscPrintf(PETSC_COMM_WORLD, "number of iterations of the method %D\n", its); >> >> PetscInt nconv; >> EPSGetConverged(eps_,&nconv); >> PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %D\n\n",nconv); >> >> if (nconv>0) { >> evals.SetSize(nconv); >> if(!rank_) { >> evecs.SetSize(matsize,nconv); >> } >> >> PetscScalar lambda; // = k >> PetscReal error,re,im; >> >> for (PetscInt i=0;i> // Get converged eigenpairs: i-th eigenvalue is stored in kr >> (real part) and >> // ki (imaginary part) >> EPSGetEigenpair(eps_,i,&lambda,PETSC_NULL,xev,PETSC_NULL); >> >> PetscScalar* tmpptr; >> >> // begin ugly inefficient(?) eigenvector gather code >> { >> VecScatter ctx; >> Vec vout; >> VecScatterCreateToZero(xev,&ctx,&vout); >> // scatter as many times as you need >> VecScatterBegin(ctx,xev,vout,INSERT_VALUES,SCATTER_FORWARD); >> VecScatterEnd(ctx,xev,vout,INSERT_VALUES,SCATTER_FORWARD); >> >> int locsize; >> VecGetLocalSize(vout,&locsize); >> VecGetArray(vout,&tmpptr); >> if(!rank_) { >> const ngla::Complex *evecptr = static_cast(tmpptr); >> >> for(size_t j = 0; j < matsize; j++) { >> evecs(j,i) = evecptr[j]; >> } >> } >> >> VecRestoreArray(xev, &tmpptr); >> // destroy scatter context and local vector when no longer needed >> VecScatterDestroy(&ctx); >> VecDestroy(&vout); >> } >> /* end ugly inefficient(?) vector gather code */ >> >> evals(i) = static_cast(lambda); >> } >> PetscPrintf(PETSC_COMM_WORLD,"\n"); >> } >> >> VecDestroy(&xev); >> EPSDestroy(&eps_); >> >> if(!nconv) >> return; >> >> if(rank_) >> evecs.SetSize(matsize,nconv); >> >> //every process already owns all the eigenvalues >> //but not all eigenstates -> only bcast all eigenstates >> MPI_Bcast(&evecs(0), nconv*matsize, MPI_DOUBLE_COMPLEX, 0, MPI_COMM_WORLD); >> } >> >> Here are some facts about my EVP: >> The dimension of the Operators A & B is ~90000. >> For nev=1 the setup time (EPSSetUp) is 18sec, and the solve time >> 11sec. (on 16 nodes) >> -> number of iterations of the method 2 >> number of converged eigenpairs: 6 >> >> For nev=150 the setup time (EPSSetUp) is 18sec, and the solve time >> 65sec. (on 16 nodes) >> -> number of iterations of the method 1 >> number of converged eigenpairs: 155 >> (The output of -log_summary is attached) >> >> >> Do you think that there is any benefit in creating the EPS object at >> the beginning of my program and only call EPSSetOperators in my Solve >> function? >> >> Regards >> Thomas >> > > Yes, you can get some modest benefit. I would strongly suggest using a parallel linear solver such as MUMPS. See section 3.4.1 of SLEPc's manual. > Jose Thx. This does wonders!! The Setup time went down from 18sec to 1 sec and the Solve time from 65sec to 10 sec. Regards Thomas > From cjm2176 at columbia.edu Thu Aug 9 07:41:43 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Thu, 09 Aug 2012 08:41:43 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> Message-ID: <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> From what I can gather from the petsc-dev source it looks like the commands in 4) will then generate the splits using strided blocks. The problem with that is the fortran code I am using (FEAP) uses petsc to assemble and solve the linear problem within its own nonlinear and time stepping schemes. The linear problem that petsc solves already has boundary conditions applied to it so petsc only sees the active (unrestrained) equations. So then in general fields can't be extracted from the active equations using strided blocks and I am stuck with generating the index sets defining the splits on my own. Will it still be possible to make use of the new DM functions in this case? FEAP website: http://www.ce.berkeley.edu/projects/feap/ Colin Quoting Matthew Knepley : > On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley wrote: > >> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe >> wrote: >> >>> Thanks all, regarding use of DM in 3.3, is the procedure now to create >>> the fields with PCFieldSplitSetIS and then use DMCreateFieldDecompositionDM >>> to create a new DM based from the new fields and the DM for the original >>> problem? >>> >> >> 1) Use petsc-dev >> >> 2) PCFieldSplitSetIS() is independent. This allows you to define splits >> however you want, but then recursive gets harder >> >> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >> apart at once >> >> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >> combination of fields, which from the command line is something like >> >> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >> > > I should have shown recursive: > > -fieldsplit_0_pc_type fieldsplit > > will split 2,0 into two blocks. > > Matt > > >> Matt >> >> >>> Colin >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From knepley at gmail.com Thu Aug 9 08:14:29 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 08:14:29 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> Message-ID: On Thu, Aug 9, 2012 at 7:41 AM, Colin McAuliffe wrote: > From what I can gather from the petsc-dev source it looks like the > commands in 4) will then generate the splits using strided blocks. The > problem with that is No. I never use strided blocks. Matt > the fortran code I am using (FEAP) uses petsc to assemble and solve the > linear problem within its own nonlinear and time stepping schemes. The > linear problem that petsc solves already has boundary conditions applied to > it so petsc only sees the active (unrestrained) equations. So then in > general fields can't be extracted from the active equations using strided > blocks and I am stuck with generating the index sets defining the splits on > my own. Will it still be possible to make use of the new DM functions in > this case? > FEAP website: > http://www.ce.berkeley.edu/**projects/feap/ > > Colin > > > Quoting Matthew Knepley : > > On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >> wrote: >> >> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe >> >wrote: >>> >>> Thanks all, regarding use of DM in 3.3, is the procedure now to create >>>> the fields with PCFieldSplitSetIS and then use >>>> DMCreateFieldDecompositionDM >>>> to create a new DM based from the new fields and the DM for the original >>>> problem? >>>> >>>> >>> 1) Use petsc-dev >>> >>> 2) PCFieldSplitSetIS() is independent. This allows you to define splits >>> however you want, but then recursive gets harder >>> >>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>> apart at once >>> >>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>> combination of fields, which from the command line is something like >>> >>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>> >>> >> I should have shown recursive: >> >> -fieldsplit_0_pc_type fieldsplit >> >> will split 2,0 into two blocks. >> >> Matt >> >> >> Matt >>> >>> >>> Colin >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Thu Aug 9 09:21:06 2012 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Thu, 09 Aug 2012 10:21:06 -0400 Subject: [petsc-users] Field split questions In-Reply-To: <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> Message-ID: <5023C752.1020401@berkeley.edu> Colin, I you block the equations in FEAP, then the restrained BCs are 'included' in assembled PETSc matrix (these dofs have rows that are zero modulo a value of unity on the diagonal and the restrained value on the right-hand side). -sg On 8/9/12 8:41 AM, Colin McAuliffe wrote: > From what I can gather from the petsc-dev source it looks like the > commands in 4) will then generate the splits using strided blocks. The > problem with that is the fortran code I am using (FEAP) uses petsc to > assemble and solve the linear problem within its own nonlinear and > time stepping schemes. The linear problem that petsc solves already > has boundary conditions applied to it so petsc only sees the active > (unrestrained) equations. So then in general fields can't be extracted > from the active equations using strided blocks and I am stuck with > generating the index sets defining the splits on my own. Will it still > be possible to make use of the new DM functions in this case? > > FEAP website: > http://www.ce.berkeley.edu/projects/feap/ > > Colin > > > Quoting Matthew Knepley : > >> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >> wrote: >> >>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe >>> wrote: >>> >>>> Thanks all, regarding use of DM in 3.3, is the procedure now to create >>>> the fields with PCFieldSplitSetIS and then use >>>> DMCreateFieldDecompositionDM >>>> to create a new DM based from the new fields and the DM for the >>>> original >>>> problem? >>>> >>> >>> 1) Use petsc-dev >>> >>> 2) PCFieldSplitSetIS() is independent. This allows you to define splits >>> however you want, but then recursive gets harder >>> >>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>> apart at once >>> >>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>> combination of fields, which from the command line is something like >>> >>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>> >> >> I should have shown recursive: >> >> -fieldsplit_0_pc_type fieldsplit >> >> will split 2,0 into two blocks. >> >> Matt >> >> >>> Matt >>> >>> >>>> Colin >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their >> experiments lead. >> -- Norbert Wiener >> > > > -- ----------------------------------------------- Sanjay Govindjee, PhD, PE Professor of Civil Engineering Vice Chair for Academic Affairs 779 Davis Hall Structural Engineering, Mechanics and Materials Department of Civil Engineering University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://www.ce.berkeley.edu/~sanjay ----------------------------------------------- From knepley at gmail.com Thu Aug 9 09:22:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 09:22:44 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <5023C752.1020401@berkeley.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> Message-ID: On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee wrote: > Colin, > I you block the equations in FEAP, then the restrained BCs are > 'included' in assembled PETSc matrix (these dofs have rows that are zero > modulo a value of unity on the diagonal and the restrained value on the > right-hand side). > However, this is not necessary with the DM interface. Matt > -sg > > On 8/9/12 8:41 AM, Colin McAuliffe wrote: > >> From what I can gather from the petsc-dev source it looks like the >> commands in 4) will then generate the splits using strided blocks. The >> problem with that is the fortran code I am using (FEAP) uses petsc to >> assemble and solve the linear problem within its own nonlinear and time >> stepping schemes. The linear problem that petsc solves already has boundary >> conditions applied to it so petsc only sees the active (unrestrained) >> equations. So then in general fields can't be extracted from the active >> equations using strided blocks and I am stuck with generating the index >> sets defining the splits on my own. Will it still be possible to make use >> of the new DM functions in this case? >> >> FEAP website: >> http://www.ce.berkeley.edu/**projects/feap/ >> >> Colin >> >> >> Quoting Matthew Knepley : >> >> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>> wrote: >>> >>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe >>> >wrote: >>>> >>>> Thanks all, regarding use of DM in 3.3, is the procedure now to create >>>>> the fields with PCFieldSplitSetIS and then use >>>>> DMCreateFieldDecompositionDM >>>>> to create a new DM based from the new fields and the DM for the >>>>> original >>>>> problem? >>>>> >>>>> >>>> 1) Use petsc-dev >>>> >>>> 2) PCFieldSplitSetIS() is independent. This allows you to define splits >>>> however you want, but then recursive gets harder >>>> >>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>>> apart at once >>>> >>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>> combination of fields, which from the command line is something like >>>> >>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>> >>>> >>> I should have shown recursive: >>> >>> -fieldsplit_0_pc_type fieldsplit >>> >>> will split 2,0 into two blocks. >>> >>> Matt >>> >>> >>> Matt >>>> >>>> >>>> Colin >>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> > -- > ------------------------------**----------------- > Sanjay Govindjee, PhD, PE > Professor of Civil Engineering > Vice Chair for Academic Affairs > > 779 Davis Hall > Structural Engineering, Mechanics and Materials > Department of Civil Engineering > University of California > Berkeley, CA 94720-1710 > > Voice: +1 510 642 6060 > FAX: +1 510 643 5264 > s_g at berkeley.edu > http://www.ce.berkeley.edu/~**sanjay > ------------------------------**----------------- > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Aug 9 09:50:09 2012 From: john.fettig at gmail.com (John Fettig) Date: Thu, 9 Aug 2012 10:50:09 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG Message-ID: I am a little confused about what Richardson means. If you use multiplicative V-cycle multigrid with Richardson KSP (and no convergence monitor), it sets the applyrichardson operator to PCApplyRichardson_MG, which appears to just run V-cycles until convergence. As far as I can tell, it doesn't ever update according to the documented x^{n+1} = x^{n} + scale*B(b - A x^{n}) If on the other hand you use full MG, it does update according to the above formula. This also happens if you set a convergence monitor. I can see how multiplicative V-cycle with Richardson is simply using multigrid as a solver. What I don't understand is how full MG with Richardson is using multigrid as a solver, because it is using the update formula above in between cycles.. Shouldn't there be a applyrichardson for full multigrid as well? If not, why? Thanks, John From cjm2176 at columbia.edu Thu Aug 9 10:02:26 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Thu, 09 Aug 2012 11:02:26 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> Message-ID: <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Sanjay, thanks for the reply but I am avoiding using blocked format since my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active on a particular node. If I use block the equations I run out of memory pretty quickly on my machine but can get to reasonable sized problems with the unblocked format. Matt, sorry I am not getting this, but I am still not sure how the DM interface works. I can see in the function PCFieldSplitSetDefaults that there is an initial call to DMCreateFieldDecomposition and subsequent calls to DMCreateSubDM based on the command line options. What I am missing is how the first call to DMCreateFieldDecomposition is able to figure out which equations belong to which field just from command line info such as -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? Thanks Colin Quoting Matthew Knepley : > On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee wrote: > >> Colin, >> I you block the equations in FEAP, then the restrained BCs are >> 'included' in assembled PETSc matrix (these dofs have rows that are zero >> modulo a value of unity on the diagonal and the restrained value on the >> right-hand side). >> > > However, this is not necessary with the DM interface. > > Matt > > >> -sg >> >> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >> >>> From what I can gather from the petsc-dev source it looks like the >>> commands in 4) will then generate the splits using strided blocks. The >>> problem with that is the fortran code I am using (FEAP) uses petsc to >>> assemble and solve the linear problem within its own nonlinear and time >>> stepping schemes. The linear problem that petsc solves already has boundary >>> conditions applied to it so petsc only sees the active (unrestrained) >>> equations. So then in general fields can't be extracted from the active >>> equations using strided blocks and I am stuck with generating the index >>> sets defining the splits on my own. Will it still be possible to make use >>> of the new DM functions in this case? >>> >>> FEAP website: >>> http://www.ce.berkeley.edu/**projects/feap/ >>> >>> Colin >>> >>> >>> Quoting Matthew Knepley : >>> >>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>> wrote: >>>> >>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe >>>> >wrote: >>>>> >>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to create >>>>>> the fields with PCFieldSplitSetIS and then use >>>>>> DMCreateFieldDecompositionDM >>>>>> to create a new DM based from the new fields and the DM for the >>>>>> original >>>>>> problem? >>>>>> >>>>>> >>>>> 1) Use petsc-dev >>>>> >>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define splits >>>>> however you want, but then recursive gets harder >>>>> >>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>>>> apart at once >>>>> >>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>> combination of fields, which from the command line is something like >>>>> >>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>> >>>>> >>>> I should have shown recursive: >>>> >>>> -fieldsplit_0_pc_type fieldsplit >>>> >>>> will split 2,0 into two blocks. >>>> >>>> Matt >>>> >>>> >>>> Matt >>>>> >>>>> >>>>> Colin >>>>>> >>>>>> -- >>>>>> Colin McAuliffe >>>>>> PhD Candidate >>>>>> Columbia University >>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> >> -- >> ------------------------------**----------------- >> Sanjay Govindjee, PhD, PE >> Professor of Civil Engineering >> Vice Chair for Academic Affairs >> >> 779 Davis Hall >> Structural Engineering, Mechanics and Materials >> Department of Civil Engineering >> University of California >> Berkeley, CA 94720-1710 >> >> Voice: +1 510 642 6060 >> FAX: +1 510 643 5264 >> s_g at berkeley.edu >> http://www.ce.berkeley.edu/~**sanjay >> ------------------------------**----------------- >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From knepley at gmail.com Thu Aug 9 10:07:10 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 10:07:10 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Message-ID: On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe wrote: > Sanjay, thanks for the reply but I am avoiding using blocked format since > my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active > on a particular node. If I use block the equations I run out of memory > pretty quickly on my machine but can get to reasonable sized problems with > the unblocked format. > > Matt, sorry I am not getting this, but I am still not sure how the DM > interface works. I can see in the function PCFieldSplitSetDefaults that > there is an initial call to DMCreateFieldDecomposition and subsequent calls > to DMCreateSubDM based on the command line options. What I am missing is > how the first call to DMCreateFieldDecomposition is able to figure out > which equations belong to which field just from command line info such as > -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? > That code is a little obscure because I am trying to be backwards compatible. I am going to remove DMCreateFieldDecompositionDM(). However, right now, what it does is call DMCreateFieldDecompositionDM() but it overrides the result with DMCreateSubDM() if command line definitions are given. Matt > Thanks > Colin > > > Quoting Matthew Knepley : > > On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >> wrote: >> >> Colin, >>> I you block the equations in FEAP, then the restrained BCs are >>> 'included' in assembled PETSc matrix (these dofs have rows that are zero >>> modulo a value of unity on the diagonal and the restrained value on the >>> right-hand side). >>> >>> >> However, this is not necessary with the DM interface. >> >> Matt >> >> >> -sg >>> >>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>> >>> From what I can gather from the petsc-dev source it looks like the >>>> commands in 4) will then generate the splits using strided blocks. The >>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>> assemble and solve the linear problem within its own nonlinear and time >>>> stepping schemes. The linear problem that petsc solves already has >>>> boundary >>>> conditions applied to it so petsc only sees the active (unrestrained) >>>> equations. So then in general fields can't be extracted from the active >>>> equations using strided blocks and I am stuck with generating the index >>>> sets defining the splits on my own. Will it still be possible to make >>>> use >>>> of the new DM functions in this case? >>>> >>>> FEAP website: >>>> http://www.ce.berkeley.edu/****projects/feap/ >>>> >>>> > >>>> >>>> Colin >>>> >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>> >>>>> wrote: >>>>> >>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>> cjm2176 at columbia.edu >>>>> >>>>>> >wrote: >>>>>> >>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>> create >>>>>> >>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>> DMCreateFieldDecompositionDM >>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>> original >>>>>>> problem? >>>>>>> >>>>>>> >>>>>>> 1) Use petsc-dev >>>>>> >>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>> splits >>>>>> however you want, but then recursive gets harder >>>>>> >>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>>>>> apart at once >>>>>> >>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>> combination of fields, which from the command line is something like >>>>>> >>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>> >>>>>> >>>>>> I should have shown recursive: >>>>> >>>>> -fieldsplit_0_pc_type fieldsplit >>>>> >>>>> will split 2,0 into two blocks. >>>>> >>>>> Matt >>>>> >>>>> >>>>> Matt >>>>> >>>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>>> >>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>> ------------------------------****----------------- >>> Sanjay Govindjee, PhD, PE >>> Professor of Civil Engineering >>> Vice Chair for Academic Affairs >>> >>> 779 Davis Hall >>> Structural Engineering, Mechanics and Materials >>> Department of Civil Engineering >>> University of California >>> Berkeley, CA 94720-1710 >>> >>> Voice: +1 510 642 6060 >>> FAX: +1 510 643 5264 >>> s_g at berkeley.edu >>> http://www.ce.berkeley.edu/~****sanjay< >>> http://www.ce.berkeley.edu/~**sanjay >>> > >>> ------------------------------****----------------- >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 10:19:00 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 15:19:00 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272AC5@exchange.scsolutions.com> Has this problem or similar problem been solved using scalapack? * Yes. We have solved A^-1 from A using scalapack. Your problem is virtually a dense problem, why choose petsc library? * Because the A' is a sparse matrix at a much larger order. What type of computer, number of cores, to be used for this problem? * Supercomputer. We are looking at about 3~5 milllion cpu hours. Jinquan First question: Are your matrices sparse or dense? * A is a dense matrix at the order of 225,000 of complex*16; How large? * 800GB on disk What type of math equations, e.g., solve Ax=b or eigenvalue problems? * A'x=b, where A'=[A^-1 U B]= sparse matrix at the order of 2 million. B = sparse matrix to be mixed and mingled with A^-1. Hong Jinquan On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong > wrote: Thanks, Prof. Zhang. I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Wednesday, August 08, 2012 3:29 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) Jinquan: You need understand some basic concepts in numerical computations, for example: - dense matrix vs. sparse matrix and their data structures - algorithms for solve Ax=b - available software packages, their targeted applications See www.mcs.anl.gov/~hzhang/teach/cs595 for a list of references. Hong What I am trying to say is that * allocated 1D array A in PETSc * passed 1D A into ScaLAPACK * obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array * indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK * constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1. I can have the last step done in using ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetUp(A);CHKERRQ(ierr); ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr); It just took too long. This is the performance issue I am having. To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, August 08, 2012 3:04 PM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong > wrote: It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc. What are you trying to say? -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Thu Aug 9 10:22:22 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Thu, 9 Aug 2012 10:22:22 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Message-ID: On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe wrote: > Sanjay, thanks for the reply but I am avoiding using blocked format since > my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active > on a particular node. If I use block the equations I run out of memory > pretty quickly on my machine but can get to reasonable sized problems with > the unblocked format. > > Matt, sorry I am not getting this, but I am still not sure how the DM > interface works. I can see in the function PCFieldSplitSetDefaults that > there is an initial call to DMCreateFieldDecomposition and subsequent calls > to DMCreateSubDM based on the command line options. What I am missing is > how the first call to DMCreateFieldDecomposition is able to figure out > which equations belong to which field just from command line info such as > -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? > This might change slightly in the near future in petsc-dev to allow one to define splits using named fields. In any event, there has to be DM support to implement the decompositions over a particular mesh/problem over that mesh. With DMDA you can essentially get combinations of strided fields in each split. DMCOMPOSITE allows you to pull out combinations of the subproblems that were put in there to begin with. If you have your own mesh, you have to write a DM implementation around it to expose the available fields. Dmitry. > > Thanks > > Colin > > > Quoting Matthew Knepley : > > On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >> wrote: >> >> Colin, >>> I you block the equations in FEAP, then the restrained BCs are >>> 'included' in assembled PETSc matrix (these dofs have rows that are zero >>> modulo a value of unity on the diagonal and the restrained value on the >>> right-hand side). >>> >>> >> However, this is not necessary with the DM interface. >> >> Matt >> >> >> -sg >>> >>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>> >>> From what I can gather from the petsc-dev source it looks like the >>>> commands in 4) will then generate the splits using strided blocks. The >>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>> assemble and solve the linear problem within its own nonlinear and time >>>> stepping schemes. The linear problem that petsc solves already has >>>> boundary >>>> conditions applied to it so petsc only sees the active (unrestrained) >>>> equations. So then in general fields can't be extracted from the active >>>> equations using strided blocks and I am stuck with generating the index >>>> sets defining the splits on my own. Will it still be possible to make >>>> use >>>> of the new DM functions in this case? >>>> >>>> FEAP website: >>>> http://www.ce.berkeley.edu/****projects/feap/ >>>> >>>> > >>>> >>>> >>>> Colin >>>> >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>> >>>>> wrote: >>>>> >>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>> cjm2176 at columbia.edu >>>>> >>>>>> >wrote: >>>>>> >>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>> create >>>>>> >>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>> DMCreateFieldDecompositionDM >>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>> original >>>>>>> problem? >>>>>>> >>>>>>> >>>>>>> 1) Use petsc-dev >>>>>> >>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>> splits >>>>>> however you want, but then recursive gets harder >>>>>> >>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all fields >>>>>> apart at once >>>>>> >>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>> combination of fields, which from the command line is something like >>>>>> >>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>> >>>>>> >>>>>> I should have shown recursive: >>>>> >>>>> -fieldsplit_0_pc_type fieldsplit >>>>> >>>>> will split 2,0 into two blocks. >>>>> >>>>> Matt >>>>> >>>>> >>>>> Matt >>>>> >>>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>>> >>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>> ------------------------------****----------------- >>> >>> Sanjay Govindjee, PhD, PE >>> Professor of Civil Engineering >>> Vice Chair for Academic Affairs >>> >>> 779 Davis Hall >>> Structural Engineering, Mechanics and Materials >>> Department of Civil Engineering >>> University of California >>> Berkeley, CA 94720-1710 >>> >>> Voice: +1 510 642 6060 >>> FAX: +1 510 643 5264 >>> s_g at berkeley.edu >>> http://www.ce.berkeley.edu/~****sanjay< >>> http://www.ce.berkeley.edu/~**sanjay >>> > >>> ------------------------------****----------------- >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 10:24:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 10:24:07 -0500 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Message-ID: On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev wrote: > > > On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe wrote: > >> Sanjay, thanks for the reply but I am avoiding using blocked format since >> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active >> on a particular node. If I use block the equations I run out of memory >> pretty quickly on my machine but can get to reasonable sized problems with >> the unblocked format. >> >> Matt, sorry I am not getting this, but I am still not sure how the DM >> interface works. I can see in the function PCFieldSplitSetDefaults that >> there is an initial call to DMCreateFieldDecomposition and subsequent calls >> to DMCreateSubDM based on the command line options. What I am missing is >> how the first call to DMCreateFieldDecomposition is able to figure out >> which equations belong to which field just from command line info such as >> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >> > This might change slightly in the near future in petsc-dev to allow one to > define splits using named fields. > In any event, there has to be DM support to implement the decompositions > over a particular mesh/problem over that mesh. > With DMDA you can essentially get combinations of strided fields in each > split. DMCOMPOSITE allows you > to pull out combinations of the subproblems that were put in there to > begin with. If you have your own mesh, you have to write > a DM implementation around it to expose the available fields. > No, No, No. You do not have to write a DM implementation. You just have to define the data layout in a PetscSection and attach it to the DM with DMSetDefaultSection. Matt > Dmitry. > >> >> Thanks >> >> Colin >> >> >> Quoting Matthew Knepley : >> >> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>> wrote: >>> >>> Colin, >>>> I you block the equations in FEAP, then the restrained BCs are >>>> 'included' in assembled PETSc matrix (these dofs have rows that are zero >>>> modulo a value of unity on the diagonal and the restrained value on the >>>> right-hand side). >>>> >>>> >>> However, this is not necessary with the DM interface. >>> >>> Matt >>> >>> >>> -sg >>>> >>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>> >>>> From what I can gather from the petsc-dev source it looks like the >>>>> commands in 4) will then generate the splits using strided blocks. The >>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>> assemble and solve the linear problem within its own nonlinear and time >>>>> stepping schemes. The linear problem that petsc solves already has >>>>> boundary >>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>> equations. So then in general fields can't be extracted from the active >>>>> equations using strided blocks and I am stuck with generating the index >>>>> sets defining the splits on my own. Will it still be possible to make >>>>> use >>>>> of the new DM functions in this case? >>>>> >>>>> FEAP website: >>>>> http://www.ce.berkeley.edu/****projects/feap/ >>>>> >>>>> > >>>>> >>>>> >>>>> Colin >>>>> >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>> >>>>>> wrote: >>>>>> >>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>> cjm2176 at columbia.edu >>>>>> >>>>>>> >wrote: >>>>>>> >>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>> create >>>>>>> >>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>> DMCreateFieldDecompositionDM >>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>> original >>>>>>>> problem? >>>>>>>> >>>>>>>> >>>>>>>> 1) Use petsc-dev >>>>>>> >>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>> splits >>>>>>> however you want, but then recursive gets harder >>>>>>> >>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>> fields >>>>>>> apart at once >>>>>>> >>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>> combination of fields, which from the command line is something like >>>>>>> >>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>> >>>>>>> >>>>>>> I should have shown recursive: >>>>>> >>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>> >>>>>> will split 2,0 into two blocks. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Matt >>>>>> >>>>>>> >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Colin McAuliffe >>>>>>>> PhD Candidate >>>>>>>> Columbia University >>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>> ------------------------------****----------------- >>>> >>>> Sanjay Govindjee, PhD, PE >>>> Professor of Civil Engineering >>>> Vice Chair for Academic Affairs >>>> >>>> 779 Davis Hall >>>> Structural Engineering, Mechanics and Materials >>>> Department of Civil Engineering >>>> University of California >>>> Berkeley, CA 94720-1710 >>>> >>>> Voice: +1 510 642 6060 >>>> FAX: +1 510 643 5264 >>>> s_g at berkeley.edu >>>> http://www.ce.berkeley.edu/~****sanjay< >>>> http://www.ce.berkeley.edu/~**sanjay >>>> > >>>> ------------------------------****----------------- >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 10:26:04 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 15:26:04 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> Jed, ? A?x=b, where A?=[A^-1 U B]= What does the notation above mean? ? That means A? consists A^-1 and B. A? is a sparse matrix at the order of 2 million. B = sparse matrix at the order of 2 million to be mixed and mingled with A^-1. What are the dimensions and sparsity of each piece. ? The dimension of A^-1 is about 225,000. B is about 2 millions. If you have an "inverse" floating around, what sort of operator is it the inverse of. ? A is a double complex matrix we have generated from a practical application using FEM. Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 9 10:28:56 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 9 Aug 2012 09:28:56 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> Message-ID: On Thu, Aug 9, 2012 at 9:26 AM, Jinquan Zhong wrote: > ? A?x=b, where A?=[A^-1 U B]= **** > > What does the notation above mean? **** > > **? **That means A? consists A^-1 and B. A? is a sparse matrix at the > order of 2 million. B = sparse matrix at the order of 2 million to be > mixed and mingled with A^-1. > > "mingled" is not a very useful term. How about using an actual equation so we can have some clue what you mean. > **** > > What are the dimensions and sparsity of each piece. **** > > **? **The dimension of A^-1 is about 225,000. B is about 2 millions. *** > * > > ** ** > > If you have an "inverse" floating around, what sort of operator is it the > inverse of.**** > > **? **A is a double complex matrix we have generated from a practical > application using FEM. > Then there is no way A^{-1} should be stored as a dense matrix. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Thu Aug 9 10:44:17 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Thu, 09 Aug 2012 11:44:17 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Message-ID: <20120809114417.p9qmipen4gc8404o@cubmail.cc.columbia.edu> > You just have to define the data layout in a PetscSection and attach it to > the DM with DMSetDefaultSection. > > Matt Ok I understand the procedure for defining the initial field layout, thanks for the help! In the documentation for 3.3p2 there is a not that PetscSectionCreate is not available in fortran, is this still the case for petsc-dev? Colin Quoting Matthew Knepley : > On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev wrote: > >> >> >> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe >> wrote: >> >>> Sanjay, thanks for the reply but I am avoiding using blocked format since >>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active >>> on a particular node. If I use block the equations I run out of memory >>> pretty quickly on my machine but can get to reasonable sized problems with >>> the unblocked format. >>> >>> Matt, sorry I am not getting this, but I am still not sure how the DM >>> interface works. I can see in the function PCFieldSplitSetDefaults that >>> there is an initial call to DMCreateFieldDecomposition and subsequent calls >>> to DMCreateSubDM based on the command line options. What I am missing is >>> how the first call to DMCreateFieldDecomposition is able to figure out >>> which equations belong to which field just from command line info such as >>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>> >> This might change slightly in the near future in petsc-dev to allow one to >> define splits using named fields. >> In any event, there has to be DM support to implement the decompositions >> over a particular mesh/problem over that mesh. >> With DMDA you can essentially get combinations of strided fields in each >> split. DMCOMPOSITE allows you >> to pull out combinations of the subproblems that were put in there to >> begin with. If you have your own mesh, you have to write >> a DM implementation around it to expose the available fields. >> > > No, No, No. You do not have to write a DM implementation. > > You just have to define the data layout in a PetscSection and attach it to > the DM with DMSetDefaultSection. > > Matt > > >> Dmitry. >> >>> >>> Thanks >>> >>> Colin >>> >>> >>> Quoting Matthew Knepley : >>> >>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>> wrote: >>>> >>>> Colin, >>>>> I you block the equations in FEAP, then the restrained BCs are >>>>> 'included' in assembled PETSc matrix (these dofs have rows that are zero >>>>> modulo a value of unity on the diagonal and the restrained value on the >>>>> right-hand side). >>>>> >>>>> >>>> However, this is not necessary with the DM interface. >>>> >>>> Matt >>>> >>>> >>>> -sg >>>>> >>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>> >>>>> From what I can gather from the petsc-dev source it looks like the >>>>>> commands in 4) will then generate the splits using strided blocks. The >>>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>>> assemble and solve the linear problem within its own nonlinear and time >>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>> boundary >>>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>>> equations. So then in general fields can't be extracted from the active >>>>>> equations using strided blocks and I am stuck with generating the index >>>>>> sets defining the splits on my own. Will it still be possible to make >>>>>> use >>>>>> of the new DM functions in this case? >>>>>> >>>>>> FEAP website: >>>>>> http://www.ce.berkeley.edu/****projects/feap/ >>>>>> >>>>>> > >>>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>> cjm2176 at columbia.edu >>>>>>> >>>>>>>> >wrote: >>>>>>>> >>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>> create >>>>>>>> >>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>> original >>>>>>>>> problem? >>>>>>>>> >>>>>>>>> >>>>>>>>> 1) Use petsc-dev >>>>>>>> >>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>> splits >>>>>>>> however you want, but then recursive gets harder >>>>>>>> >>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>> fields >>>>>>>> apart at once >>>>>>>> >>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>> combination of fields, which from the command line is something like >>>>>>>> >>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>> >>>>>>>> >>>>>>>> I should have shown recursive: >>>>>>> >>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>> >>>>>>> will split 2,0 into two blocks. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Colin McAuliffe >>>>>>>>> PhD Candidate >>>>>>>>> Columbia University >>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>> ------------------------------****----------------- >>>>> >>>>> Sanjay Govindjee, PhD, PE >>>>> Professor of Civil Engineering >>>>> Vice Chair for Academic Affairs >>>>> >>>>> 779 Davis Hall >>>>> Structural Engineering, Mechanics and Materials >>>>> Department of Civil Engineering >>>>> University of California >>>>> Berkeley, CA 94720-1710 >>>>> >>>>> Voice: +1 510 642 6060 >>>>> FAX: +1 510 643 5264 >>>>> s_g at berkeley.edu >>>>> http://www.ce.berkeley.edu/~****sanjay< >>>>> http://www.ce.berkeley.edu/~**sanjay >>>>> > >>>>> ------------------------------****----------------- >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From knepley at gmail.com Thu Aug 9 10:49:04 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 10:49:04 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120809114417.p9qmipen4gc8404o@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120809114417.p9qmipen4gc8404o@cubmail.cc.columbia.edu> Message-ID: On Thu, Aug 9, 2012 at 10:44 AM, Colin McAuliffe wrote: > You just have to define the data layout in a PetscSection and attach it to >> the DM with DMSetDefaultSection. >> >> Matt >> > > Ok I understand the procedure for defining the initial field layout, > thanks for the help! In the documentation for 3.3p2 there is a not that > PetscSectionCreate is not available in fortran, is this still the case for > petsc-dev? > I will put all that in this weekend (or Friday if I can get to it). Chris Eldred has also asked. Matt > Colin > > > > Quoting Matthew Knepley : > > On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >> wrote: >> >> >>> >>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe >> >wrote: >>> >>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>> since >>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>> active >>>> on a particular node. If I use block the equations I run out of memory >>>> pretty quickly on my machine but can get to reasonable sized problems >>>> with >>>> the unblocked format. >>>> >>>> Matt, sorry I am not getting this, but I am still not sure how the DM >>>> interface works. I can see in the function PCFieldSplitSetDefaults that >>>> there is an initial call to DMCreateFieldDecomposition and subsequent >>>> calls >>>> to DMCreateSubDM based on the command line options. What I am missing is >>>> how the first call to DMCreateFieldDecomposition is able to figure out >>>> which equations belong to which field just from command line info such >>>> as >>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>>> >>>> This might change slightly in the near future in petsc-dev to allow >>> one to >>> define splits using named fields. >>> In any event, there has to be DM support to implement the decompositions >>> over a particular mesh/problem over that mesh. >>> With DMDA you can essentially get combinations of strided fields in each >>> split. DMCOMPOSITE allows you >>> to pull out combinations of the subproblems that were put in there to >>> begin with. If you have your own mesh, you have to write >>> a DM implementation around it to expose the available fields. >>> >>> >> No, No, No. You do not have to write a DM implementation. >> >> You just have to define the data layout in a PetscSection and attach it to >> the DM with DMSetDefaultSection. >> >> Matt >> >> >> Dmitry. >>> >>> >>>> Thanks >>>> >>>> Colin >>>> >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>> >>>>> wrote: >>>>> >>>>> Colin, >>>>> >>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>> 'included' in assembled PETSc matrix (these dofs have rows that are >>>>>> zero >>>>>> modulo a value of unity on the diagonal and the restrained value on >>>>>> the >>>>>> right-hand side). >>>>>> >>>>>> >>>>>> However, this is not necessary with the DM interface. >>>>> >>>>> Matt >>>>> >>>>> >>>>> -sg >>>>> >>>>>> >>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>> >>>>>> From what I can gather from the petsc-dev source it looks like the >>>>>> >>>>>>> commands in 4) will then generate the splits using strided blocks. >>>>>>> The >>>>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>>>> assemble and solve the linear problem within its own nonlinear and >>>>>>> time >>>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>>> boundary >>>>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>>>> equations. So then in general fields can't be extracted from the >>>>>>> active >>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>> index >>>>>>> sets defining the splits on my own. Will it still be possible to make >>>>>>> use >>>>>>> of the new DM functions in this case? >>>>>>> >>>>>>> FEAP website: >>>>>>> http://www.ce.berkeley.edu/******projects/feap/ >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> Quoting Matthew Knepley : >>>>>>> >>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>>> > >>>>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>> cjm2176 at columbia.edu >>>>>>>> >>>>>>>> >wrote: >>>>>>>>> >>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>>> create >>>>>>>>> >>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>>> original >>>>>>>>>> problem? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1) Use petsc-dev >>>>>>>>>> >>>>>>>>> >>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>>> splits >>>>>>>>> however you want, but then recursive gets harder >>>>>>>>> >>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>> fields >>>>>>>>> apart at once >>>>>>>>> >>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>>> combination of fields, which from the command line is something >>>>>>>>> like >>>>>>>>> >>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>> >>>>>>>>> >>>>>>>>> I should have shown recursive: >>>>>>>>> >>>>>>>> >>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>> >>>>>>>> will split 2,0 into two blocks. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Colin McAuliffe >>>>>>>>>> PhD Candidate >>>>>>>>>> Columbia University >>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>> which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>> ------------------------------******----------------- >>>>>> >>>>>> Sanjay Govindjee, PhD, PE >>>>>> Professor of Civil Engineering >>>>>> Vice Chair for Academic Affairs >>>>>> >>>>>> 779 Davis Hall >>>>>> Structural Engineering, Mechanics and Materials >>>>>> Department of Civil Engineering >>>>>> University of California >>>>>> Berkeley, CA 94720-1710 >>>>>> >>>>>> Voice: +1 510 642 6060 >>>>>> FAX: +1 510 643 5264 >>>>>> s_g at berkeley.edu >>>>>> http://www.ce.berkeley.edu/~******sanjay >>>>>> >>>>>> >< >>>>>> http://www.ce.berkeley.edu/~****sanjay >>>>>> >>>>>> > >>>>>> > >>>>>> ------------------------------******----------------- >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 10:50:43 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 15:50:43 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> ? A?x=b, where A?=[A^-1 U B]= What does the notation above mean? ? That means A? consists A^-1 and B. A? is a sparse matrix at the order of 2 million. B = sparse matrix at the order of 2 million to be mixed and mingled with A^-1. "mingled" is not a very useful term. How about using an actual equation so we can have some clue what you mean. o I agree. It is hard to have an accurate equation to for the structure of A?. But if you are familiar with FEM, you may know global stiffness matrix K=SIGMA_ij( K_ij ), the assembling process of elemental stiffness. Here A?=K. o We partition the FEM domain into two physical pieces due to different operations they are defined for. After completing each operation, we need to operate on A^-1_ij and B_i?j? such that we can have K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?). If you have an "inverse" floating around, what sort of operator is it the inverse of. ? A is a double complex matrix we have generated from a practical application using FEM. Then there is no way A^{-1} should be stored as a dense matrix. o It is stored as a dense matrix in scalapack. It stays in the same way in cores as scalapack finishes its inversion. We could define A^-1 as dense matrix in PETSc. o Another constraint we have is we have to use direct solver for QA purposes. This brings up the needs to convert the dense matrix into MATAIJ or MATMPIAIJ. However, in PETSc, converting dense matrix-formatted A^-1 into MATAIJ/ MATMPIAIJ also takes lots of time. Here is what I am looking at after all these discussion: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ? Some function similar to MatCreateMPIAIJWithArrays(..,A^-1) with the additional capability to specify the location of upper-left corner of distributed A^-1 in A?, the global matrix. ? The reason is that in CSR format defaulted in PETSc, the entire row has to be input in one rank. In this application, we have partitioned this row into different pieces due to the operation on ScaLAPACK. Due to large amount of data, there is no practical way that we can gather the entire portioned rows of A^-1 into on rank. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 9 11:41:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 9 Aug 2012 10:41:23 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> Message-ID: On Thu, Aug 9, 2012 at 9:50 AM, Jinquan Zhong wrote: > > > > ** > > "mingled" is not a very useful term. How about using an actual equation so > we can have some clue what you mean.**** > > **o **I agree. It is hard to have an accurate equation to for the > structure of A?. But if you are familiar with FEM, you may know global > stiffness matrix K=SIGMA_ij( K_ij ), the assembling process of elemental > stiffness. Here A?=K. > This cannot be true because you say A' is the larger part sparse and part dense matrix. Maybe you mean A rather than A'? > ** ** > > **o **We partition the FEM domain into two physical pieces due to > different operations they are defined for. After completing each > operation, we need to operate on A^-1_ij and B_i?j? such that we can have > K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?). > The sum of the inverses are not the inverses of the sum. Can you back up further to explain what underlying problem you are trying to solve? > **** > > **** > > ** ** > > If you have an "inverse" floating around, what sort of operator is it the > inverse of.**** > > ? A is a double complex matrix we have generated from a practical > application using FEM. **** > > Then there is no way A^{-1} should be stored as a dense matrix. **** > > **o **It is stored as a dense matrix in scalapack. It stays in the > same way in cores as scalapack finishes its inversion. We could define > A^-1 as dense matrix in PETSc. > It should not be done this way. A^{-1} should not be stored explicitly. Store the sparse finite element matrix A. Then when you want to "apply A^{-1}", solve with the sparse matrix. > **** > > **o **Another constraint we have is we have to use direct solver for QA > purposes. This brings up the needs to convert the dense matrix into MATAIJ > or MATMPIAIJ. However, in PETSc, converting dense matrix-formatted A^-1 > into MATAIJ/ MATMPIAIJ also takes lots of time. > Choose the right tool for the job. When I cook dinner tonight, I plan to use a stove and a frying pan instead of a dump truck radiator and 200 feet of 18 gauge copper wire woven into a basket. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 12:02:04 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 17:02:04 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> "mingled" is not a very useful term. How about using an actual equation so we can have some clue what you mean. o I agree. It is hard to have an accurate equation to for the structure of A?. But if you are familiar with FEM, you may know global stiffness matrix K=SIGMA_ij( K_ij ), the assembling process of elemental stiffness. Here A?=K. This cannot be true because you say A' is the larger part sparse and part dense matrix. Maybe you mean A rather than A'? **Jed, A?=[A^-1 U B], not the transpose of A. We need to solve for A?*x=b. A is dense matrix. B is a sparse matrix. Therefore, A? is a sparse matrix when the rule K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?) applies. o We partition the FEM domain into two physical pieces due to different operations they are defined for. After completing each operation, we need to operate on A^-1_ij and B_i?j? such that we can have K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?). The sum of the inverses are not the inverses of the sum. Can you back up further to explain what underlying problem you are trying to solve? ** As I mentioned A?=K is the assembled global matrix using the rule K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?) from FEM. Here, SIGMA_ij (A^-1_ij) denotes the assembling process for A^-1_ij into K according the DOFs of each node in the FEM model, so does SIGMA_i?j?(B_i?j?). in this application, A^-1 and B have different indices sets. Some of them may locate at the same locations in K. If you have an "inverse" floating around, what sort of operator is it the inverse of. ? A is a double complex matrix we have generated from a practical application using FEM. Then there is no way A^{-1} should be stored as a dense matrix. o It is stored as a dense matrix in scalapack. It stays in the same way in cores as scalapack finishes its inversion. We could define A^-1 as dense matrix in PETSc. It should not be done this way. A^{-1} should not be stored explicitly. Store the sparse finite element matrix A. Then when you want to "apply A^{-1}", solve with the sparse matrix. ** Jed, you are getting close to understand the problem related to QA. A has to be inverted explicitly. A^-1 has to be known entry by entry such that each entry in B could be assembled with A^-1 to form A?=K. This is a requirement beyond technical issue. This is a QA issue. o Another constraint we have is we have to use direct solver for QA purposes. This brings up the needs to convert the dense matrix into MATAIJ or MATMPIAIJ. However, in PETSc, converting dense matrix-formatted A^-1 into MATAIJ/ MATMPIAIJ also takes lots of time. Choose the right tool for the job. When I cook dinner tonight, I plan to use a stove and a frying pan instead of a dump truck radiator and 200 feet of 18 gauge copper wire woven into a basket. ** That why I came here for. PETSc is the right tool for solver in the last step A?*x=b. However, due to the differences in addressing the matrix between PETSc and ScaLAPACK, I am asking for the kind help from this community to current issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 9 12:11:39 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 9 Aug 2012 11:11:39 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> Message-ID: On Thu, Aug 9, 2012 at 11:02 AM, Jinquan Zhong wrote: > o I agree. It is hard to have an accurate equation to for the > structure of A?. But if you are familiar with FEM, you may know global > stiffness matrix K=SIGMA_ij( K_ij ), the assembling process of elemental > stiffness. Here A?=K. **** > > This cannot be true because you say A' is the larger part sparse and part > dense matrix. Maybe you mean A rather than A'?**** > > ** ** > > **Jed, A?=[A^-1 U B], not the transpose of A. > I understand that it's not the transpose. I still don't have a clue what the notation [A^{-1} U B] means. I would think it means some block decomposed thing, but I don't think you mean just adding columns or taking a product of matrices. Using some standard mathematical notation would help. > We need to solve for A?*x=b. A is dense matrix. > Where does A come from? Why is it dense? > B is a sparse matrix. Therefore, A? is a sparse matrix when the rule K=A?=SIGMA_ij > (A^-1_ij)+SIGMA_i?j?(B_i?j?) applies.**** > > **** > > o We partition the FEM domain into two physical pieces due to > different operations they are defined for. After completing each > operation, we need to operate on A^-1_ij and B_i?j? such that we can have > K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?).**** > > The sum of the inverses are not the inverses of the sum. Can you back up > further to explain what underlying problem you are trying to solve?**** > > ** ** > > ** As I mentioned A?=K is the assembled global matrix using the rule K=A?=SIGMA_ij > (A^-1_ij)+SIGMA_i?j?(B_i?j?) from FEM. Here, SIGMA_ij (A^-1_ij) denotes > the assembling process for A^-1_ij into K according the DOFs of each node > in the FEM model, > Is A^{-1} an operator on some subdomain? Are you trying to implement a substructuring algorithm? What is B physically? > so does SIGMA_i?j?(B_i?j?). in this application, A^-1 and B have > different indices sets. Some of them may locate at the same locations in K. > **** > > **** > > **** > > If you have an "inverse" floating around, what sort of operator is it the > inverse of.**** > > ? A is a double complex matrix we have generated from a practical > application using FEM. **** > > Then there is no way A^{-1} should be stored as a dense matrix. **** > > o It is stored as a dense matrix in scalapack. It stays in the same > way in cores as scalapack finishes its inversion. We could define A^-1 as > dense matrix in PETSc. **** > > It should not be done this way. A^{-1} should not be stored explicitly. > Store the sparse finite element matrix A. Then when you want to "apply > A^{-1}", solve with the sparse matrix.**** > > ** ** > > ** Jed, you are getting close to understand the problem related to QA. > A has to be inverted explicitly. A^-1 has to be known entry by entry such > that each entry in B could be assembled with A^-1 to form A?=K. This is a > requirement beyond technical issue. This is a QA issue. > What does QA stand for? Can you explain why B needs the entries of A^{-1}? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 12:28:45 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 17:28:45 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> Hope this is not too technical. ? **Jed, A?=[A^-1 U B], not the transpose of A. I understand that it's not the transpose. I still don't have a clue what the notation [A^{-1} U B] means. I would think it means some block decomposed thing, but I don't think you mean just adding columns or taking a product of matrices. Using some standard mathematical notation would help. >> A?=[A^-1 U B] means matrix A? consists all elements from A^-1, the inverted A, and all elements from B. One simple equation is K=[Kst] =A? = ASSEMBLE_ij (A^-1_ij) + ASSEMBLE_i?j?(B_i?j?), here (s,t), (i,j) and (i?,j?) = different index sets i.e., entry K_st= A^-1_st+ B_st, = 0 when A^-1_st=0 and B_st=0 = A^-1_st when B_st =0 = B_st when A^-1_st =0 We need to solve for A?*x=b. A is dense matrix. Where does A come from? Why is it dense? >> A comes from wave propagation. It is dense since it denotes a function of Green function. ** As I mentioned A?=K is the assembled global matrix using the rule K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?) from FEM. Here, SIGMA_ij (A^-1_ij) denotes the assembling process for A^-1_ij into K according the DOFs of each node in the FEM model, Is A^{-1} an operator on some subdomain? Are you trying to implement a substructuring algorithm? What is B physically? >> NO. A^{-1} denotes the inverted A. B is a sparse matrix of much larger order. Then there is no way A^{-1} should be stored as a dense matrix. o It is stored as a dense matrix in scalapack. It stays in the same way in cores as scalapack finishes its inversion. We could define A^-1 as dense matrix in PETSc. It should not be done this way. A^{-1} should not be stored explicitly. Store the sparse finite element matrix A. Then when you want to "apply A^{-1}", solve with the sparse matrix. ** Jed, you are getting close to understand the problem related to QA. A has to be inverted explicitly. A^-1 has to be known entry by entry such that each entry in B could be assembled with A^-1 to form A?=K. This is a requirement beyond technical issue. This is a QA issue. What does QA stand for? Can you explain why B needs the entries of A^{-1}? >> QA means quality assurance. It is a procedure to ensure product quality. In Eq. K=A?=ASSEMBLE_ij (A^-1_ij)+ ASSEMBLE_i?j?(B_i?j?) Entry B_i?j? and A^-1_ij may or may not locate at the same row and col. That why we need explicitly each entry in B_i?j? and A^-1_ij to assemble K. The big picture is that K is the final sparse matrix we need to solve K*x=A?*x=b. However, K indexed by (s,t) needs to be constructed in terms of dense matrix A and sparse matrix B using index sets (i,j) and (i?,j?). -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Aug 9 13:20:32 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 9 Aug 2012 13:20:32 -0500 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> Message-ID: "QA means quality assurance.", which has nothing to do with the technical discussions here. However, I take this opportunity to promote http://aspiritech.org, an non-profit organization with a mission to provide employment on *QA* for high functioning individuals on the Autism Spectrum. Hong On Thu, Aug 9, 2012 at 12:28 PM, Jinquan Zhong wrote: > Hope this is not too technical. J**** > > ** ** > > **Jed, A?=[A^-1 U B], not the transpose of A.**** > > ** ** > > I understand that it's not the transpose. I still don't have a clue what > the notation [A^{-1} U B] means. I would think it means some block > decomposed thing, but I don't think you mean just adding columns or taking > a product of matrices. Using some standard mathematical notation would help. > **** > > ** ** > > >> A?=[A^-1 U B] means matrix A? consists all elements from A^-1, the > inverted A, and all elements from B. One simple equation is **** > > ** ** > > K=[Kst] =A? = ASSEMBLE_ij (A^-1_ij) + ASSEMBLE_i?j?(B_i?j?), > here (s,t), (i,j) and (i?,j?) = different index sets**** > > ** ** > > i.e., entry K_st= A^-1_st+ B_st, **** > > = 0 when A^-1_st=0 and B_st=0**** > > = A^-1_st when B_st =0**** > > = B_st when A^-1_st =0**** > > > **** > > We need to solve for A?*x=b. A is dense matrix.**** > > ** ** > > Where does A come from? Why is it dense?**** > > ** ** > > >> A comes from wave propagation. It is dense since it denotes a function > of Green function.**** > > **** > > **** > > ** As I mentioned A?=K is the assembled global matrix using the rule > K=A?=SIGMA_ij (A^-1_ij)+SIGMA_i?j?(B_i?j?) from FEM. Here, SIGMA_ij > (A^-1_ij) denotes the assembling process for A^-1_ij into K according the > DOFs of each node in the FEM model,**** > > ** ** > > Is A^{-1} an operator on some subdomain? Are you trying to implement a > substructuring algorithm? What is B physically?**** > > ** ** > > >> NO. A^{-1} denotes the inverted A. B is a sparse matrix of much > larger order.**** > > **** > > Then there is no way A^{-1} should be stored as a dense matrix. **** > > o It is stored as a dense matrix in scalapack. It stays in the same > way in cores as scalapack finishes its inversion. We could define A^-1 as > dense matrix in PETSc. **** > > It should not be done this way. A^{-1} should not be stored explicitly. > Store the sparse finite element matrix A. Then when you want to "apply > A^{-1}", solve with the sparse matrix.**** > > ** ** > > ** Jed, you are getting close to understand the problem related to QA. > A has to be inverted explicitly. A^-1 has to be known entry by entry such > that each entry in B could be assembled with A^-1 to form A?=K. This is a > requirement beyond technical issue. This is a QA issue.**** > > ** ** > > What does QA stand for? Can you explain why B needs the entries of A^{-1}? > **** > > ** ** > > >> QA means quality assurance. It is a procedure to ensure product > quality. In Eq. **** > > ** ** > > K=A?=ASSEMBLE_ij (A^-1_ij)+ ASSEMBLE_i?j?(B_i?j?)**** > > **** > > Entry B_i?j? and A^-1_ij may or may not locate at the same row and col. > That why we need explicitly each entry in B_i?j? and A^-1_ij to assemble > K. The big picture is that K is the final sparse matrix we need to solve > K*x=A?*x=b. However, K indexed by (s,t) needs to be constructed in terms > of dense matrix A and sparse matrix B using index sets (i,j) and (i?,j?). > **** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 9 13:25:45 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 9 Aug 2012 18:25:45 +0000 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F272B7F@exchange.scsolutions.com> Well, what I was trying to say is the need to get into the scalapack. Obviously, this need brings up the technical query on a proper PETSc function to build sparse matrix from a dense matrix of large size previously partitioned beyond the setting in PETSc. Otherwise, I would take another route to construct the matrix for PETSc. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Thursday, August 09, 2012 11:21 AM To: PETSc users list Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...) "QA means quality assurance.", which has nothing to do with the technical discussions here. However, I take this opportunity to promote http://aspiritech.org, an non-profit organization with a mission to provide employment on QA for high functioning individuals on the Autism Spectrum. Hong On Thu, Aug 9, 2012 at 12:28 PM, Jinquan Zhong > wrote: Hope this is not too technical. :) **Jed, A'=[A^-1 U B], not the transpose of A. I understand that it's not the transpose. I still don't have a clue what the notation [A^{-1} U B] means. I would think it means some block decomposed thing, but I don't think you mean just adding columns or taking a product of matrices. Using some standard mathematical notation would help. >> A'=[A^-1 U B] means matrix A' consists all elements from A^-1, the inverted A, and all elements from B. One simple equation is K=[Kst] =A' = ASSEMBLE_ij (A^-1_ij) + ASSEMBLE_i'j'(B_i'j'), here (s,t), (i,j) and (i',j') = different index sets i.e., entry K_st= A^-1_st+ B_st, = 0 when A^-1_st=0 and B_st=0 = A^-1_st when B_st =0 = B_st when A^-1_st =0 We need to solve for A'*x=b. A is dense matrix. Where does A come from? Why is it dense? >> A comes from wave propagation. It is dense since it denotes a function of Green function. ** As I mentioned A'=K is the assembled global matrix using the rule K=A'=SIGMA_ij (A^-1_ij)+SIGMA_i'j'(B_i'j') from FEM. Here, SIGMA_ij (A^-1_ij) denotes the assembling process for A^-1_ij into K according the DOFs of each node in the FEM model, Is A^{-1} an operator on some subdomain? Are you trying to implement a substructuring algorithm? What is B physically? >> NO. A^{-1} denotes the inverted A. B is a sparse matrix of much larger order. Then there is no way A^{-1} should be stored as a dense matrix. o It is stored as a dense matrix in scalapack. It stays in the same way in cores as scalapack finishes its inversion. We could define A^-1 as dense matrix in PETSc. It should not be done this way. A^{-1} should not be stored explicitly. Store the sparse finite element matrix A. Then when you want to "apply A^{-1}", solve with the sparse matrix. ** Jed, you are getting close to understand the problem related to QA. A has to be inverted explicitly. A^-1 has to be known entry by entry such that each entry in B could be assembled with A^-1 to form A'=K. This is a requirement beyond technical issue. This is a QA issue. What does QA stand for? Can you explain why B needs the entries of A^{-1}? >> QA means quality assurance. It is a procedure to ensure product quality. In Eq. K=A'=ASSEMBLE_ij (A^-1_ij)+ ASSEMBLE_i'j'(B_i'j') Entry B_i'j' and A^-1_ij may or may not locate at the same row and col. That why we need explicitly each entry in B_i'j' and A^-1_ij to assemble K. The big picture is that K is the final sparse matrix we need to solve K*x=A'*x=b. However, K indexed by (s,t) needs to be constructed in terms of dense matrix A and sparse matrix B using index sets (i,j) and (i',j'). -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexvg77 at gmail.com Thu Aug 9 15:19:15 2012 From: alexvg77 at gmail.com (Alexander Goncharov) Date: Thu, 09 Aug 2012 13:19:15 -0700 Subject: [petsc-users] RAM usage by a sparse matrix Message-ID: <1344543555.20884.21.camel@DeathStar> Hello PETSC developers! I would like to ask a couple of questions about symmetric sparse matrix storage 1. Is matrix frees unused preallocated nonzero space. For example, when run with option -info, it says how many nonzeros were unused. Is this unused space unallocated? 2. How PETSC stores symmetric sparse matrices on a GPU? Does it stores only half of it or not? Thank you! Alexander. From yifli82 at gmail.com Thu Aug 9 17:20:00 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 18:20:00 -0400 Subject: [petsc-users] create a sparse symmetric matrix Message-ID: Hi all, I only filled the entries in the upper triangular part of the matrix, however, I got 0 when I tried to get the value of some entry in the lower triangular part. Here's how I do it: MatCreate(PETSC_COMM_WORLD, &A); MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); MatSetType(A, MATSEQSBAIJ); MatSetUp(A); // I set the value for the entry at (0, 2) int i = 2, j = 0; MatGetValues(A,1,&i,1,&j,values); Any help is highly appreicated. Yifei -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 9 17:37:59 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 9 Aug 2012 17:37:59 -0500 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: Message-ID: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> John, On Aug 9, 2012, at 9:50 AM, John Fettig wrote: > I am a little confused about what Richardson means. If you use > multiplicative V-cycle multigrid with Richardson KSP (and no > convergence monitor), it sets the applyrichardson operator to > PCApplyRichardson_MG, which appears to just run V-cycles until > convergence. Yes, this is correct. > As far as I can tell, it doesn't ever update according > to the documented > > x^{n+1} = x^{n} + scale*B(b - A x^{n}) > In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. > If on the other hand you use full MG, it does update according to the > above formula. This also happens if you set a convergence monitor. > > I can see how multiplicative V-cycle with Richardson is simply using > multigrid as a solver. What I don't understand is how full MG with > Richardson is using multigrid as a solver, because it is using the > update formula above in between cycles.. Shouldn't there be a > applyrichardson for full multigrid as well? If not, why? I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. Barry > > Thanks, > John From yifli82 at gmail.com Thu Aug 9 17:39:20 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 18:39:20 -0400 Subject: [petsc-users] vector outer product Message-ID: Hi, I did not find any function doing this in the manual. Could someone suggest a good way to do this? Thanks Yifei -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pinna.2006 at gmail.com Thu Aug 9 18:18:26 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 01:18:26 +0200 Subject: [petsc-users] Configure error In-Reply-To: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: I followed your advice and I installed the binary version of MPICH for win 32 bit. Than I prepared the envirnmental variable with PETSC_DIR=`pwd`; export PETSC_DIR then in the petsc directory ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' --download-f-blas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ \(x86\)/MPICH2 and I got the error "C compiler you provided -with-cc win32fe cl does not work" Part of the log look like this sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -c -o conftest.o conftest.c Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -c -o conftest.o conftest.c sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN conftest.c sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -c -o conftest.o conftest.c Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -c -o conftest.o conftest.c sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN conftest.c Pushing language C Popping language C Pushing language Cxx Popping language Cxx Pushing language FC Popping language FC Pushing language C Popping language C sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -o conftest.exe conftest.o Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl -o conftest.exe conftest.o sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN LINK : fatal error LNK1104: cannot open file 'kernel32.lib' Possible ERROR while running linker: output: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN LINK : fatal error LNK1104: cannot open file 'kernel32.lib' ret = 512 so it look like a directory problem, but the installation is standard, maybe I cannot use it with MS Visual C++ 2008 Express? Thank you for your time. Fabio 2012/8/6 Barry Smith : > > On Aug 5, 2012, at 5:19 PM, Fabio wrote: > >> Thank you for your quick reply,I will try with a reboot. >> >> Regarding the compiler my idea was to use MSVC express 2008. I thought >> it was possible to compile both library with gcc under cygwin. >> From your reply I guess this is not the case. >> Does this mean I should compile Petsc with MSVC express as well, isn't it? > > Yes, you need to compile PETSc with the compiler you would like to use and make sure to use the MPICH binary install since that works with the Microsoft compilers. > > Barry > >> >> >> Fabio. >> >> >> >> 2012/8/5 Satish Balay : >>> On Sun, 5 Aug 2012, Fabio wrote: >>> >>>> Dear all, >>>> I am new to petsc, and I've been struggling with it for the last days. >>>> >>>> I want to install it on Windows 7, and first I tried to install the >>>> Microsoft MPI and then compiling the MPICH. They both seems ok but I >>>> am not able to compile Petsc with any of them. >>>> I am mostly interested to MPICH (for portability). >>>> I used this script to configure >>>> >>>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 >>>> --download-f2cblaslapack=1 >>>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 >>> >>> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? >>> >>> [MS MPI is for use with MS compiler only. MPICH binary install is for >>> MS compilers, and MPICH source install is for cygwin gcc/g++] >>> >>>> >>>> and I've got this error, >>>> >>>> UNABLE to EXECUTE BINARIES for config/configure.py >>>> --------------------------------------------------------------------------------------- >>>> [Errno 16] Device or resource busy: 'conftest.exe' >>>> >>>> This error seems to happen pretty randomly at differente places in the script. >>>> Any idea to solve this problem? >>> >>> Its bad file locks on windows. The only suggestion we have is reboot. >>> >>> For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be >>> that issue. >>> >>> If you still have issues - send the complete log [and logfile] to petsc-maint. >>> >>> Satish >>> >>>> Thank you all >>>> >>>> Fabio >>>> > From fabio.pinna.2006 at gmail.com Thu Aug 9 18:24:09 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 01:24:09 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: I forgot to say that I start cygwin from the MSVC 2008 Expresse prompt and that running cl does produce some output (basically prviding an help for the cl function) 2012/8/10 Fabio : > I followed your advice and I installed the binary version of MPICH for > win 32 bit. > Than I prepared the envirnmental variable with > PETSC_DIR=`pwd`; export PETSC_DIR > > then in the petsc directory > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' > --download-f-blas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > \(x86\)/MPICH2 > > and I got the error > > "C compiler you provided -with-cc win32fe cl does not work" > > Part of the log look like this > > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > -c -o conftest.o conftest.c > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > cl -c -o conftest.o conftest.c > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > conftest.c > > > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > -c -o conftest.o conftest.c > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > cl -c -o conftest.o conftest.c > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > conftest.c > > > Pushing language C > Popping language C > Pushing language Cxx > Popping language Cxx > Pushing language FC > Popping language FC > Pushing language C > Popping language C > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > -o conftest.exe conftest.o > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > cl -o conftest.exe conftest.o > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > > > Possible ERROR while running linker: output: Warning: win32fe: Path > Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > > ret = 512 > > > > so it look like a directory problem, but the installation is standard, > maybe I cannot use it with MS Visual C++ 2008 Express? > Thank you for your time. > Fabio > > > 2012/8/6 Barry Smith : >> >> On Aug 5, 2012, at 5:19 PM, Fabio wrote: >> >>> Thank you for your quick reply,I will try with a reboot. >>> >>> Regarding the compiler my idea was to use MSVC express 2008. I thought >>> it was possible to compile both library with gcc under cygwin. >>> From your reply I guess this is not the case. >>> Does this mean I should compile Petsc with MSVC express as well, isn't it? >> >> Yes, you need to compile PETSc with the compiler you would like to use and make sure to use the MPICH binary install since that works with the Microsoft compilers. >> >> Barry >> >>> >>> >>> Fabio. >>> >>> >>> >>> 2012/8/5 Satish Balay : >>>> On Sun, 5 Aug 2012, Fabio wrote: >>>> >>>>> Dear all, >>>>> I am new to petsc, and I've been struggling with it for the last days. >>>>> >>>>> I want to install it on Windows 7, and first I tried to install the >>>>> Microsoft MPI and then compiling the MPICH. They both seems ok but I >>>>> am not able to compile Petsc with any of them. >>>>> I am mostly interested to MPICH (for portability). >>>>> I used this script to configure >>>>> >>>>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 >>>>> --download-f2cblaslapack=1 >>>>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 >>>> >>>> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? >>>> >>>> [MS MPI is for use with MS compiler only. MPICH binary install is for >>>> MS compilers, and MPICH source install is for cygwin gcc/g++] >>>> >>>>> >>>>> and I've got this error, >>>>> >>>>> UNABLE to EXECUTE BINARIES for config/configure.py >>>>> --------------------------------------------------------------------------------------- >>>>> [Errno 16] Device or resource busy: 'conftest.exe' >>>>> >>>>> This error seems to happen pretty randomly at differente places in the script. >>>>> Any idea to solve this problem? >>>> >>>> Its bad file locks on windows. The only suggestion we have is reboot. >>>> >>>> For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be >>>> that issue. >>>> >>>> If you still have issues - send the complete log [and logfile] to petsc-maint. >>>> >>>> Satish >>>> >>>>> Thank you all >>>>> >>>>> Fabio >>>>> >> From balay at mcs.anl.gov Thu Aug 9 18:46:32 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 9 Aug 2012 18:46:32 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: Why use this very old version petsc-2.3.3? If this old version is really needed - you might want to copy over bin/win32fe/* files over from the latest version. > --with-mpi-dir=/cygdrive/c/Program\ Files\ \(x86\)/MPICH2 I don't remember if it will handle current mpi - so might want to use --with-mpi-include/--with-mpi-lib options. Also the braces in the PATH can potentially cause problem. Satish On Fri, 10 Aug 2012, Fabio wrote: > I forgot to say that I start cygwin from the MSVC 2008 Expresse prompt > and that running cl does produce some output (basically prviding an > help for the cl function) > > 2012/8/10 Fabio : > > I followed your advice and I installed the binary version of MPICH for > > win 32 bit. > > Than I prepared the envirnmental variable with > > PETSC_DIR=`pwd`; export PETSC_DIR > > > > then in the petsc directory > > > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' > > --download-f-blas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > > \(x86\)/MPICH2 > > > > and I got the error > > > > "C compiler you provided -with-cc win32fe cl does not work" > > > > Part of the log look like this > > > > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > > -c -o conftest.o conftest.c > > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > > cl -c -o conftest.o conftest.c > > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > > > conftest.c > > > > > > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > > -c -o conftest.o conftest.c > > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > > cl -c -o conftest.o conftest.c > > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > > > conftest.c > > > > > > Pushing language C > > Popping language C > > Pushing language Cxx > > Popping language Cxx > > Pushing language FC > > Popping language FC > > Pushing language C > > Popping language C > > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > > -o conftest.exe conftest.o > > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > > cl -o conftest.exe conftest.o > > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > > > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > > > > > > Possible ERROR while running linker: output: Warning: win32fe: Path > > Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > > > > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > > > > ret = 512 > > > > > > > > so it look like a directory problem, but the installation is standard, > > maybe I cannot use it with MS Visual C++ 2008 Express? > > Thank you for your time. > > Fabio > > > > > > 2012/8/6 Barry Smith : > >> > >> On Aug 5, 2012, at 5:19 PM, Fabio wrote: > >> > >>> Thank you for your quick reply,I will try with a reboot. > >>> > >>> Regarding the compiler my idea was to use MSVC express 2008. I thought > >>> it was possible to compile both library with gcc under cygwin. > >>> From your reply I guess this is not the case. > >>> Does this mean I should compile Petsc with MSVC express as well, isn't it? > >> > >> Yes, you need to compile PETSc with the compiler you would like to use and make sure to use the MPICH binary install since that works with the Microsoft compilers. > >> > >> Barry > >> > >>> > >>> > >>> Fabio. > >>> > >>> > >>> > >>> 2012/8/5 Satish Balay : > >>>> On Sun, 5 Aug 2012, Fabio wrote: > >>>> > >>>>> Dear all, > >>>>> I am new to petsc, and I've been struggling with it for the last days. > >>>>> > >>>>> I want to install it on Windows 7, and first I tried to install the > >>>>> Microsoft MPI and then compiling the MPICH. They both seems ok but I > >>>>> am not able to compile Petsc with any of them. > >>>>> I am mostly interested to MPICH (for portability). > >>>>> I used this script to configure > >>>>> > >>>>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 > >>>>> --download-f2cblaslapack=1 > >>>>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 > >>>> > >>>> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? > >>>> > >>>> [MS MPI is for use with MS compiler only. MPICH binary install is for > >>>> MS compilers, and MPICH source install is for cygwin gcc/g++] > >>>> > >>>>> > >>>>> and I've got this error, > >>>>> > >>>>> UNABLE to EXECUTE BINARIES for config/configure.py > >>>>> --------------------------------------------------------------------------------------- > >>>>> [Errno 16] Device or resource busy: 'conftest.exe' > >>>>> > >>>>> This error seems to happen pretty randomly at differente places in the script. > >>>>> Any idea to solve this problem? > >>>> > >>>> Its bad file locks on windows. The only suggestion we have is reboot. > >>>> > >>>> For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be > >>>> that issue. > >>>> > >>>> If you still have issues - send the complete log [and logfile] to petsc-maint. > >>>> > >>>> Satish > >>>> > >>>>> Thank you all > >>>>> > >>>>> Fabio > >>>>> > >> > From knepley at gmail.com Thu Aug 9 19:16:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 19:16:47 -0500 Subject: [petsc-users] RAM usage by a sparse matrix In-Reply-To: <1344543555.20884.21.camel@DeathStar> References: <1344543555.20884.21.camel@DeathStar> Message-ID: On Thu, Aug 9, 2012 at 3:19 PM, Alexander Goncharov wrote: > Hello PETSC developers! > > I would like to ask a couple of questions about symmetric sparse matrix > storage > > 1. Is matrix frees unused preallocated nonzero space. For example, when > run with option -info, it says how many nonzeros were unused. Is this > unused space unallocated? > Yes, during assembly. > 2. How PETSC stores symmetric sparse matrices on a GPU? Does it stores > only half of it or not? > We do not have symmetric storage on the GPU since we only have what the CUDA libraries have. Matt > Thank you! > > Alexander. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexvg77 at gmail.com Thu Aug 9 19:19:42 2012 From: alexvg77 at gmail.com (Alexander Goncharov) Date: Thu, 09 Aug 2012 17:19:42 -0700 Subject: [petsc-users] RAM usage by a sparse matrix In-Reply-To: References: <1344543555.20884.21.camel@DeathStar> Message-ID: <1344557982.27168.1.camel@deathstar> Thank you, Matt! On Thu, 2012-08-09 at 19:16 -0500, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 3:19 PM, Alexander Goncharov > wrote: > Hello PETSC developers! > > I would like to ask a couple of questions about symmetric > sparse matrix > storage > > 1. Is matrix frees unused preallocated nonzero space. For > example, when > run with option -info, it says how many nonzeros were unused. > Is this > unused space unallocated? > > > Yes, during assembly. > > 2. How PETSC stores symmetric sparse matrices on a GPU? Does > it stores > only half of it or not? > > > We do not have symmetric storage on the GPU since we only have what > the CUDA libraries have. > > > Matt > > Thank you! > > Alexander. > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Thu Aug 9 19:45:13 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 19:45:13 -0500 Subject: [petsc-users] create a sparse symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 5:20 PM, Yifei Li wrote: > Hi all, > > I only filled the entries in the upper triangular part of the matrix, > however, I got 0 when I tried to get the value of some entry in the lower > triangular part. > > Here's how I do it: > > > MatCreate(PETSC_COMM_WORLD, &A); > > MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); > > MatSetType(A, MATSEQSBAIJ); > > MatSetUp(A); > > // I set the value for the entry at (0, 2) > > int i = 2, j = 0; > > MatGetValues(A,1,&i,1,&j,values); > MatGetValues() is not synchronous, so it would not be able to communicate to the correct process to retrieve the values. Thus, you cannot retrieve the lower triangle. This is analogous to only having access to local values with VecGetArray(). Matt > Any help is highly appreicated. > > > > Yifei > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 20:15:41 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 20:15:41 -0500 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: > Hi, > > I did not find any function doing this in the manual. Could someone > suggest a good way to do this? > That gives a dense matrix. You would never want to do this. You would keep it in unassembled form. Matt > > Thanks > > Yifei > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Thu Aug 9 21:53:17 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 22:53:17 -0400 Subject: [petsc-users] create a sparse symmetric matrix In-Reply-To: References: Message-ID: Matt, Thanks. I have another two related questions: 1) So just because I did not get the correct value doesn't mean the symmetric matrix is not setup correctly? 2) Which function should I use to retrieve the value then? I can't get the correct value even though I disabled MPI when building PETSC? Yifei On Thu, Aug 9, 2012 at 8:45 PM, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 5:20 PM, Yifei Li wrote: > >> Hi all, >> >> I only filled the entries in the upper triangular part of the matrix, >> however, I got 0 when I tried to get the value of some entry in the lower >> triangular part. >> >> Here's how I do it: >> >> >> MatCreate(PETSC_COMM_WORLD, &A); >> >> MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); >> >> MatSetType(A, MATSEQSBAIJ); >> >> MatSetUp(A); >> >> // I set the value for the entry at (0, 2) >> >> int i = 2, j = 0; >> >> MatGetValues(A,1,&i,1,&j,values); >> > MatGetValues() is not synchronous, so it would not be able to communicate > to the > correct process to retrieve the values. Thus, you cannot retrieve the > lower triangle. > This is analogous to only having access to local values with VecGetArray(). > > Matt > > >> Any help is highly appreicated. >> >> >> >> Yifei >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Thu Aug 9 21:56:05 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 22:56:05 -0400 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: Matt, I'm new to PETSC, can you pls explain what you mean by 'keep it in unassembled form'? A small example would be nice too. Thanks Yifei On Thu, Aug 9, 2012 at 9:15 PM, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: > >> Hi, >> >> I did not find any function doing this in the manual. Could someone >> suggest a good way to do this? >> > > That gives a dense matrix. You would never want to do this. You would keep > it in unassembled form. > > Matt > > >> >> Thanks >> >> Yifei >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 21:59:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 21:59:23 -0500 Subject: [petsc-users] create a sparse symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 9:53 PM, Yifei Li wrote: > Matt, > > Thanks. I have another two related questions: > 1) So just because I did not get the correct value doesn't mean the > symmetric matrix is not setup correctly? > Right. > 2) Which function should I use to retrieve the value then? I can't get the > correct value even though I disabled MPI when building PETSC? > Since it would require communication, there is no easy way to do this. Why would you need this? Matt > Yifei > > > > On Thu, Aug 9, 2012 at 8:45 PM, Matthew Knepley wrote: > >> On Thu, Aug 9, 2012 at 5:20 PM, Yifei Li wrote: >> >>> Hi all, >>> >>> I only filled the entries in the upper triangular part of the matrix, >>> however, I got 0 when I tried to get the value of some entry in the lower >>> triangular part. >>> >>> Here's how I do it: >>> >>> >>> MatCreate(PETSC_COMM_WORLD, &A); >>> >>> MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); >>> >>> MatSetType(A, MATSEQSBAIJ); >>> >>> MatSetUp(A); >>> >>> // I set the value for the entry at (0, 2) >>> >>> int i = 2, j = 0; >>> >>> MatGetValues(A,1,&i,1,&j,values); >>> >> MatGetValues() is not synchronous, so it would not be able to communicate >> to the >> correct process to retrieve the values. Thus, you cannot retrieve the >> lower triangle. >> This is analogous to only having access to local values with >> VecGetArray(). >> >> Matt >> >> >>> Any help is highly appreicated. >>> >>> >>> >>> Yifei >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 21:59:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 21:59:55 -0500 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 9:56 PM, Yifei Li wrote: > Matt, > > I'm new to PETSC, can you pls explain what you mean by 'keep it in > unassembled form'? > Just keep two vectors, u^T v, instead of the dense matrix. Matt > A small example would be nice too. Thanks > > Yifei > > On Thu, Aug 9, 2012 at 9:15 PM, Matthew Knepley wrote: > >> On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: >> >>> Hi, >>> >>> I did not find any function doing this in the manual. Could someone >>> suggest a good way to do this? >>> >> >> That gives a dense matrix. You would never want to do this. You would >> keep it in unassembled form. >> >> Matt >> >> >>> >>> Thanks >>> >>> Yifei >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Thu Aug 9 22:09:38 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 23:09:38 -0400 Subject: [petsc-users] create a sparse symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 10:59 PM, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 9:53 PM, Yifei Li wrote: > >> Matt, >> >> Thanks. I have another two related questions: >> 1) So just because I did not get the correct value doesn't mean the >> symmetric matrix is not setup correctly? >> > > Right. > > >> 2) Which function should I use to retrieve the value then? I can't get >> the correct value even though I disabled MPI when building PETSC? >> > > Since it would require communication, there is no easy way to do this. Why > would you need this? > > I just want to make sure the lower triangular part gets the same values as the upper triangular part. > > > Matt > > >> Yifei >> >> >> >> On Thu, Aug 9, 2012 at 8:45 PM, Matthew Knepley wrote: >> >>> On Thu, Aug 9, 2012 at 5:20 PM, Yifei Li wrote: >>> >>>> Hi all, >>>> >>>> I only filled the entries in the upper triangular part of the matrix, >>>> however, I got 0 when I tried to get the value of some entry in the lower >>>> triangular part. >>>> >>>> Here's how I do it: >>>> >>>> >>>> MatCreate(PETSC_COMM_WORLD, &A); >>>> >>>> MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); >>>> >>>> MatSetType(A, MATSEQSBAIJ); >>>> >>>> MatSetUp(A); >>>> >>>> // I set the value for the entry at (0, 2) >>>> >>>> int i = 2, j = 0; >>>> >>>> MatGetValues(A,1,&i,1,&j,values); >>>> >>> MatGetValues() is not synchronous, so it would not be able to >>> communicate to the >>> correct process to retrieve the values. Thus, you cannot retrieve the >>> lower triangle. >>> This is analogous to only having access to local values with >>> VecGetArray(). >>> >>> Matt >>> >>> >>>> Any help is highly appreicated. >>>> >>>> >>>> >>>> Yifei >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Thu Aug 9 22:19:33 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 23:19:33 -0400 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 10:59 PM, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 9:56 PM, Yifei Li wrote: > >> Matt, >> >> I'm new to PETSC, can you pls explain what you mean by 'keep it in >> unassembled form'? >> > > Just keep two vectors, u^T v, instead of the dense matrix. > > But I need the result of the outer product (e.g, A) for further calculation (e.g., A*W). > > > Matt > > >> A small example would be nice too. Thanks >> >> Yifei >> >> On Thu, Aug 9, 2012 at 9:15 PM, Matthew Knepley wrote: >> >>> On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: >>> >>>> Hi, >>>> >>>> I did not find any function doing this in the manual. Could someone >>>> suggest a good way to do this? >>>> >>> >>> That gives a dense matrix. You would never want to do this. You would >>> keep it in unassembled form. >>> >>> Matt >>> >>> >>>> >>>> Thanks >>>> >>>> Yifei >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 22:20:40 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 22:20:40 -0500 Subject: [petsc-users] create a sparse symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 10:09 PM, Yifei Li wrote: > > > On Thu, Aug 9, 2012 at 10:59 PM, Matthew Knepley wrote: > >> On Thu, Aug 9, 2012 at 9:53 PM, Yifei Li wrote: >> >>> Matt, >>> >>> Thanks. I have another two related questions: >>> 1) So just because I did not get the correct value doesn't mean the >>> symmetric matrix is not setup correctly? >>> >> >> Right. >> >> >>> 2) Which function should I use to retrieve the value then? I can't get >>> the correct value even though I disabled MPI when building PETSC? >>> >> >> Since it would require communication, there is no easy way to do this. >> Why would you need this? >> >> > I just want to make sure the lower triangular part gets the same values as > the upper triangular part. > That is what View() is for. Matt > >> >> Matt >> >> >>> Yifei >>> >>> >>> >>> On Thu, Aug 9, 2012 at 8:45 PM, Matthew Knepley wrote: >>> >>>> On Thu, Aug 9, 2012 at 5:20 PM, Yifei Li wrote: >>>> >>>>> Hi all, >>>>> >>>>> I only filled the entries in the upper triangular part of the matrix, >>>>> however, I got 0 when I tried to get the value of some entry in the lower >>>>> triangular part. >>>>> >>>>> Here's how I do it: >>>>> >>>>> >>>>> MatCreate(PETSC_COMM_WORLD, &A); >>>>> >>>>> MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n); >>>>> >>>>> MatSetType(A, MATSEQSBAIJ); >>>>> >>>>> MatSetUp(A); >>>>> >>>>> // I set the value for the entry at (0, 2) >>>>> >>>>> int i = 2, j = 0; >>>>> >>>>> MatGetValues(A,1,&i,1,&j,values); >>>>> >>>> MatGetValues() is not synchronous, so it would not be able to >>>> communicate to the >>>> correct process to retrieve the values. Thus, you cannot retrieve the >>>> lower triangle. >>>> This is analogous to only having access to local values with >>>> VecGetArray(). >>>> >>>> Matt >>>> >>>> >>>>> Any help is highly appreicated. >>>>> >>>>> >>>>> >>>>> Yifei >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 22:22:35 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 22:22:35 -0500 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 10:19 PM, Yifei Li wrote: > > > On Thu, Aug 9, 2012 at 10:59 PM, Matthew Knepley wrote: > >> On Thu, Aug 9, 2012 at 9:56 PM, Yifei Li wrote: >> >>> Matt, >>> >>> I'm new to PETSC, can you pls explain what you mean by 'keep it in >>> unassembled form'? >>> >> >> Just keep two vectors, u^T v, instead of the dense matrix. >> >> > But I need the result of the outer product (e.g, A) for > further calculation (e.g., A*W). > A * W = A * (u v^T) = (A * u) v^T Matt > >> >> Matt >> >> >>> A small example would be nice too. Thanks >>> >>> Yifei >>> >>> On Thu, Aug 9, 2012 at 9:15 PM, Matthew Knepley wrote: >>> >>>> On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: >>>> >>>>> Hi, >>>>> >>>>> I did not find any function doing this in the manual. Could someone >>>>> suggest a good way to do this? >>>>> >>>> >>>> That gives a dense matrix. You would never want to do this. You would >>>> keep it in unassembled form. >>>> >>>> Matt >>>> >>>> >>>>> >>>>> Thanks >>>>> >>>>> Yifei >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Thu Aug 9 22:26:29 2012 From: yifli82 at gmail.com (Yifei Li) Date: Thu, 9 Aug 2012 23:26:29 -0400 Subject: [petsc-users] vector outer product In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 11:22 PM, Matthew Knepley wrote: > On Thu, Aug 9, 2012 at 10:19 PM, Yifei Li wrote: > >> >> >> On Thu, Aug 9, 2012 at 10:59 PM, Matthew Knepley wrote: >> >>> On Thu, Aug 9, 2012 at 9:56 PM, Yifei Li wrote: >>> >>>> Matt, >>>> >>>> I'm new to PETSC, can you pls explain what you mean by 'keep it in >>>> unassembled form'? >>>> >>> >>> Just keep two vectors, u^T v, instead of the dense matrix. >>> >>> >> But I need the result of the outer product (e.g, A) for >> further calculation (e.g., A*W). >> > > A * W = A * (u v^T) = (A * u) v^T > > Ah, how come I missed such a simple technique thanks :) > > > Matt > > >> >>> >>> Matt >>> >>> >>>> A small example would be nice too. Thanks >>>> >>>> Yifei >>>> >>>> On Thu, Aug 9, 2012 at 9:15 PM, Matthew Knepley wrote: >>>> >>>>> On Thu, Aug 9, 2012 at 5:39 PM, Yifei Li wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I did not find any function doing this in the manual. Could someone >>>>>> suggest a good way to do this? >>>>>> >>>>> >>>>> That gives a dense matrix. You would never want to do this. You would >>>>> keep it in unassembled form. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Yifei >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siaeleni at hotmail.com Thu Aug 9 23:36:01 2012 From: siaeleni at hotmail.com (Eleni Siampli) Date: Fri, 10 Aug 2012 07:36:01 +0300 Subject: [petsc-users] symmetric matrix Message-ID: Hello all, Lets say that I have a square Mat A, nxn and is upper triangular. The rest elements are 0.for example: lets say that I have a matrix 3x3 : [5 7 8;0 9 1;0 0 2]What I want is to take the symmetric matrix: [5 7 8;7 9 1;8 1 2]I think that I have to use MatCreateSeqSBAIJ(), but I don't understand exactly how can I use it.What is the block? Thank you in advance,Helen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 23:39:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 23:39:01 -0500 Subject: [petsc-users] symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 11:36 PM, Eleni Siampli wrote: > Hello all, > > Lets say that I have a square Mat A, nxn and is upper triangular. The rest > elements are 0. > for example: lets say that I have a matrix 3x3 : [5 7 8;0 9 1;0 0 2] > What I want is to take the symmetric matrix: [5 7 8;7 9 1;8 1 2] > I think that I have to use MatCreateSeqSBAIJ(), but I don't understand > exactly how can I use it. > What is the block? > Set blocksize to 1 and you should have what you want. Matt > Thank you in advance, > Helen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From siaeleni at hotmail.com Thu Aug 9 23:45:05 2012 From: siaeleni at hotmail.com (Eleni Siampli) Date: Fri, 10 Aug 2012 07:45:05 +0300 Subject: [petsc-users] symmetric matrix In-Reply-To: References: , Message-ID: What about nz and nnz? Thank you for the quick answer Helen On Thu, Aug 9, 2012 at 11:36 PM, Eleni Siampli wrote: Hello all, Lets say that I have a square Mat A, nxn and is upper triangular. The rest elements are 0.for example: lets say that I have a matrix 3x3 : [5 7 8;0 9 1;0 0 2]What I want is to take the symmetric matrix: [5 7 8;7 9 1;8 1 2] I think that I have to use MatCreateSeqSBAIJ(), but I don't understand exactly how can I use it.What is the block? Set blocksize to 1 and you should have what you want. Matt Thank you in advance,Helen -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 9 23:52:26 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Aug 2012 23:52:26 -0500 Subject: [petsc-users] symmetric matrix In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 11:45 PM, Eleni Siampli wrote: > > What about nz and nnz? > These apply only to the upper triangular part. Matt > Thank you for the quick answer > > Helen > > On Thu, Aug 9, 2012 at 11:36 PM, Eleni Siampli wrote: > > Hello all, > > Lets say that I have a square Mat A, nxn and is upper triangular. The rest > elements are 0. > for example: lets say that I have a matrix 3x3 : [5 7 8;0 9 1;0 0 2] > What I want is to take the symmetric matrix: [5 7 8;7 9 1;8 1 2] > I think that I have to use MatCreateSeqSBAIJ(), but I don't understand > exactly how can I use it. > What is the block? > > > Set blocksize to 1 and you should have what you want. > > Matt > > > Thank you in advance, > Helen > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 10 00:11:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 9 Aug 2012 23:11:58 -0600 Subject: [petsc-users] Customizeing MatSetValuesBlocked(...) In-Reply-To: <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F2726EC@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F27270B@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272936@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272955@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272975@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729AB@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F2729FA@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A0A@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272A19@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272ACF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272AF5@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B55@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F272B6C@exchange.scsolutions.com> Message-ID: On Thu, Aug 9, 2012 at 11:28 AM, Jinquan Zhong wrote: > Hope this is not too technical. J**** > > ** ** > > **Jed, A?=[A^-1 U B], not the transpose of A.**** > > ** ** > > I understand that it's not the transpose. I still don't have a clue what > the notation [A^{-1} U B] means. I would think it means some block > decomposed thing, but I don't think you mean just adding columns or taking > a product of matrices. Using some standard mathematical notation would help. > **** > > ** ** > > >> A?=[A^-1 U B] means matrix A? consists all elements from A^-1, the > inverted A, and all elements from B. One simple equation is **** > > ** ** > > K=[Kst] =A? = ASSEMBLE_ij (A^-1_ij) + ASSEMBLE_i?j?(B_i?j?), > here (s,t), (i,j) and (i?,j?) = different index sets**** > > ** ** > > i.e., entry K_st= A^-1_st+ B_st, **** > > = 0 when A^-1_st=0 and B_st=0**** > > = A^-1_st when B_st =0**** > > = B_st when A^-1_st =0 > Oh, so U meant "union" (in some funny sense). We would often write things like K = \sum_i R_i^T A_i^{-1} R_i + B where R_i is a restriction to "part" i. The great thing about mathematical notation is that we can communicate precisely what we mean. In this case, we would have been able to ask how the restriction operators R_i are connected (e.g. how much they overlap). > **** > > > **** > > We need to solve for A?*x=b. A is dense matrix.**** > > ** ** > > Where does A come from? Why is it dense?**** > > ** ** > > >> A comes from wave propagation. It is dense since it denotes a function > of Green function. > What is the kernel? Is it a pure integral operator (a discretized first kind Fredholm integral equation), identity plus a compact integral operator (a second kind Fredholm integral operator), or something else? This matters a great deal. > Is A^{-1} an operator on some subdomain? Are you trying to implement a > substructuring algorithm? What is B physically?**** > > ** ** > > >> NO. A^{-1} denotes the inverted A. B is a sparse matrix of much > larger order. > What is B physically? In what way is it coupled to A? What does QA stand for? Can you explain why B needs the entries of A^{-1}?** > ** > > ** ** > > >> QA means quality assurance. It is a procedure to ensure product > quality. In Eq. **** > > ** ** > > K=A?=ASSEMBLE_ij (A^-1_ij)+ ASSEMBLE_i?j?(B_i?j?)**** > > **** > > Entry B_i?j? and A^-1_ij may or may not locate at the same row and col. > That why we need explicitly each entry in B_i?j? and A^-1_ij to assemble > K. The big picture is that K is the final sparse matrix we need to solve > K*x=A?*x=b. However, K indexed by (s,t) needs to be constructed in terms > of dense matrix A and sparse matrix B using index sets (i,j) and (i?,j?). > **** > It is very likely that you can (and should) solve K x = b without computing explicit entries for K. You would apply its action by some other procedure, likely not involving explicit construction of the A_i^{-1} matrices either. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pinna.2006 at gmail.com Fri Aug 10 07:24:34 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 14:24:34 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: I updated to the last version of Petsc then I used again PETSC_DIR=`pwd`; export PETSC_DIR then in the petsc directory ./configure --with-cc='win32fe cl' --with-fc=0 --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ \(x86\)/MPICH2 Now I have an error, "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" The directory anyway is there and cygwin could easily access it Also trying with mpi-include/lib gives the same error. What can I do? Fabio 2012/8/10 Satish Balay : > Why use this very old version petsc-2.3.3? > > If this old version is really needed - you might want to copy over > bin/win32fe/* files over from the latest version. > >> --with-mpi-dir=/cygdrive/c/Program\ Files\ \(x86\)/MPICH2 > > I don't remember if it will handle current mpi - so might want to use > --with-mpi-include/--with-mpi-lib options. Also the braces in the PATH > can potentially cause problem. > > Satish > > On Fri, 10 Aug 2012, Fabio wrote: > >> I forgot to say that I start cygwin from the MSVC 2008 Expresse prompt >> and that running cl does produce some output (basically prviding an >> help for the cl function) >> >> 2012/8/10 Fabio : >> > I followed your advice and I installed the binary version of MPICH for >> > win 32 bit. >> > Than I prepared the envirnmental variable with >> > PETSC_DIR=`pwd`; export PETSC_DIR >> > >> > then in the petsc directory >> > >> > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' >> > --download-f-blas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ >> > \(x86\)/MPICH2 >> > >> > and I got the error >> > >> > "C compiler you provided -with-cc win32fe cl does not work" >> > >> > Part of the log look like this >> > >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl >> > -c -o conftest.o conftest.c >> > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe >> > cl -c -o conftest.o conftest.c >> > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN >> > >> > conftest.c >> > >> > >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl >> > -c -o conftest.o conftest.c >> > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe >> > cl -c -o conftest.o conftest.c >> > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN >> > >> > conftest.c >> > >> > >> > Pushing language C >> > Popping language C >> > Pushing language Cxx >> > Popping language Cxx >> > Pushing language FC >> > Popping language FC >> > Pushing language C >> > Popping language C >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl >> > -o conftest.exe conftest.o >> > Executing: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe >> > cl -o conftest.exe conftest.o >> > sh: Warning: win32fe: Path Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN >> > >> > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' >> > >> > >> > Possible ERROR while running linker: output: Warning: win32fe: Path >> > Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN >> > >> > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' >> > >> > ret = 512 >> > >> > >> > >> > so it look like a directory problem, but the installation is standard, >> > maybe I cannot use it with MS Visual C++ 2008 Express? >> > Thank you for your time. >> > Fabio >> > >> > >> > 2012/8/6 Barry Smith : >> >> >> >> On Aug 5, 2012, at 5:19 PM, Fabio wrote: >> >> >> >>> Thank you for your quick reply,I will try with a reboot. >> >>> >> >>> Regarding the compiler my idea was to use MSVC express 2008. I thought >> >>> it was possible to compile both library with gcc under cygwin. >> >>> From your reply I guess this is not the case. >> >>> Does this mean I should compile Petsc with MSVC express as well, isn't it? >> >> >> >> Yes, you need to compile PETSc with the compiler you would like to use and make sure to use the MPICH binary install since that works with the Microsoft compilers. >> >> >> >> Barry >> >> >> >>> >> >>> >> >>> Fabio. >> >>> >> >>> >> >>> >> >>> 2012/8/5 Satish Balay : >> >>>> On Sun, 5 Aug 2012, Fabio wrote: >> >>>> >> >>>>> Dear all, >> >>>>> I am new to petsc, and I've been struggling with it for the last days. >> >>>>> >> >>>>> I want to install it on Windows 7, and first I tried to install the >> >>>>> Microsoft MPI and then compiling the MPICH. They both seems ok but I >> >>>>> am not able to compile Petsc with any of them. >> >>>>> I am mostly interested to MPICH (for portability). >> >>>>> I used this script to configure >> >>>>> >> >>>>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 >> >>>>> --download-f2cblaslapack=1 >> >>>>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 >> >>>> >> >>>> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? >> >>>> >> >>>> [MS MPI is for use with MS compiler only. MPICH binary install is for >> >>>> MS compilers, and MPICH source install is for cygwin gcc/g++] >> >>>> >> >>>>> >> >>>>> and I've got this error, >> >>>>> >> >>>>> UNABLE to EXECUTE BINARIES for config/configure.py >> >>>>> --------------------------------------------------------------------------------------- >> >>>>> [Errno 16] Device or resource busy: 'conftest.exe' >> >>>>> >> >>>>> This error seems to happen pretty randomly at differente places in the script. >> >>>>> Any idea to solve this problem? >> >>>> >> >>>> Its bad file locks on windows. The only suggestion we have is reboot. >> >>>> >> >>>> For some cygwin issues - you need to run 'rebaseall' but this doesn;t apper to be >> >>>> that issue. >> >>>> >> >>>> If you still have issues - send the complete log [and logfile] to petsc-maint. >> >>>> >> >>>> Satish >> >>>> >> >>>>> Thank you all >> >>>>> >> >>>>> Fabio >> >>>>> >> >> >> From koki.imada at york.ac.uk Fri Aug 10 07:57:39 2012 From: koki.imada at york.ac.uk (Koki Imada) Date: Fri, 10 Aug 2012 13:57:39 +0100 Subject: [petsc-users] Using PetscBarrier or MPI_barrier in Fortran 90 Message-ID: <50250543.8080306@york.ac.uk> Hello, I'm fairly new to PETSc/MPI programming, and I'm stuck with the use of barriers. I have a code of the following form: 0. Initialise PETSc... 1. Processor 1 (or the "root") reports to user (using "print" statement). 2. "Root" does some work, while all other processors do nothing. 3. "Root" allocates array "A". 4. All processors does some work on array "A". 5. "Root" deallocates array "A". 6. PETSc is finalised. In order for this to work, I would like to put barriers between steps 3&4, and steps 4&5, and also between steps 2&3 (optional). Simply putting "call MPI_barrier(MPI_comm_world,ierr)" between the steps doesn't do anything and I know this doesn't work due to the lack of communication between the processors (or otherwise), but I have never been able to find the "correct" way to implement the barrier... If somebody could show me how to correctly implement the barrier (either MPI or PETSc) between the steps as described above, I will be most grateful. Best Regards, Koki From knepley at gmail.com Fri Aug 10 08:28:48 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Aug 2012 08:28:48 -0500 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: On Fri, Aug 10, 2012 at 7:24 AM, Fabio wrote: > I updated to the last version of Petsc then I used again > > PETSC_DIR=`pwd`; export PETSC_DIR > > then in the petsc directory > ./configure --with-cc='win32fe cl' --with-fc=0 > --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > \(x86\)/MPICH2 > > Now I have an error, > "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" > > The directory anyway is there and cygwin could easily access it > > Also trying with mpi-include/lib gives the same error. > What can I do? > Always send configure.log with a bug report. The path should work, but in order to test, can you make a link to a path without spaces and parentheses? ln -s /cygdrive/c/Program\ Files\ \(x86\)/MPICH2 /cygdrive/c/MPICH2 Matt > > > Fabio > > > > > > > 2012/8/10 Satish Balay : > > Why use this very old version petsc-2.3.3? > > > > If this old version is really needed - you might want to copy over > > bin/win32fe/* files over from the latest version. > > > >> --with-mpi-dir=/cygdrive/c/Program\ Files\ \(x86\)/MPICH2 > > > > I don't remember if it will handle current mpi - so might want to use > > --with-mpi-include/--with-mpi-lib options. Also the braces in the PATH > > can potentially cause problem. > > > > Satish > > > > On Fri, 10 Aug 2012, Fabio wrote: > > > >> I forgot to say that I start cygwin from the MSVC 2008 Expresse prompt > >> and that running cl does produce some output (basically prviding an > >> help for the cl function) > >> > >> 2012/8/10 Fabio : > >> > I followed your advice and I installed the binary version of MPICH for > >> > win 32 bit. > >> > Than I prepared the envirnmental variable with > >> > PETSC_DIR=`pwd`; export PETSC_DIR > >> > > >> > then in the petsc directory > >> > > >> > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' > >> > --download-f-blas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > >> > \(x86\)/MPICH2 > >> > > >> > and I got the error > >> > > >> > "C compiler you provided -with-cc win32fe cl does not work" > >> > > >> > Part of the log look like this > >> > > >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > >> > -c -o conftest.o conftest.c > >> > Executing: > /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > >> > cl -c -o conftest.o conftest.c > >> > sh: Warning: win32fe: Path Not Found: > C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > >> > > >> > conftest.c > >> > > >> > > >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > >> > -c -o conftest.o conftest.c > >> > Executing: > /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > >> > cl -c -o conftest.o conftest.c > >> > sh: Warning: win32fe: Path Not Found: > C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > >> > > >> > conftest.c > >> > > >> > > >> > Pushing language C > >> > Popping language C > >> > Pushing language Cxx > >> > Popping language Cxx > >> > Pushing language FC > >> > Popping language FC > >> > Pushing language C > >> > Popping language C > >> > sh: /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe cl > >> > -o conftest.exe conftest.o > >> > Executing: > /cygdrive/d/Workspace/PETSc/petsc-2.3.3-p13/bin/win32fe/win32fe > >> > cl -o conftest.exe conftest.o > >> > sh: Warning: win32fe: Path Not Found: > C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > >> > > >> > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > >> > > >> > > >> > Possible ERROR while running linker: output: Warning: win32fe: Path > >> > Not Found: C:\PROGRA~2\MICROS~1.0\SharedIDE\BIN > >> > > >> > LINK : fatal error LNK1104: cannot open file 'kernel32.lib' > >> > > >> > ret = 512 > >> > > >> > > >> > > >> > so it look like a directory problem, but the installation is standard, > >> > maybe I cannot use it with MS Visual C++ 2008 Express? > >> > Thank you for your time. > >> > Fabio > >> > > >> > > >> > 2012/8/6 Barry Smith : > >> >> > >> >> On Aug 5, 2012, at 5:19 PM, Fabio > wrote: > >> >> > >> >>> Thank you for your quick reply,I will try with a reboot. > >> >>> > >> >>> Regarding the compiler my idea was to use MSVC express 2008. I > thought > >> >>> it was possible to compile both library with gcc under cygwin. > >> >>> From your reply I guess this is not the case. > >> >>> Does this mean I should compile Petsc with MSVC express as well, > isn't it? > >> >> > >> >> Yes, you need to compile PETSc with the compiler you would like > to use and make sure to use the MPICH binary install since that works with > the Microsoft compilers. > >> >> > >> >> Barry > >> >> > >> >>> > >> >>> > >> >>> Fabio. > >> >>> > >> >>> > >> >>> > >> >>> 2012/8/5 Satish Balay : > >> >>>> On Sun, 5 Aug 2012, Fabio wrote: > >> >>>> > >> >>>>> Dear all, > >> >>>>> I am new to petsc, and I've been struggling with it for the last > days. > >> >>>>> > >> >>>>> I want to install it on Windows 7, and first I tried to install > the > >> >>>>> Microsoft MPI and then compiling the MPICH. They both seems ok > but I > >> >>>>> am not able to compile Petsc with any of them. > >> >>>>> I am mostly interested to MPICH (for portability). > >> >>>>> I used this script to configure > >> >>>>> > >> >>>>> ./config/configure.py PETSC_ARCH=win32-gnu --with-fc=0 > >> >>>>> --download-f2cblaslapack=1 > >> >>>>> ?with-mpi-dir=/cygdrive/d/Workspace/MPICH/mpich2-1.4.1p1 > >> >>>> > >> >>>> What compilers do you plan to use? MS C/C++ or cygwin gcc/g++? > >> >>>> > >> >>>> [MS MPI is for use with MS compiler only. MPICH binary install is > for > >> >>>> MS compilers, and MPICH source install is for cygwin gcc/g++] > >> >>>> > >> >>>>> > >> >>>>> and I've got this error, > >> >>>>> > >> >>>>> UNABLE to EXECUTE BINARIES for > config/configure.py > >> >>>>> > --------------------------------------------------------------------------------------- > >> >>>>> [Errno 16] Device or resource busy: 'conftest.exe' > >> >>>>> > >> >>>>> This error seems to happen pretty randomly at differente places > in the script. > >> >>>>> Any idea to solve this problem? > >> >>>> > >> >>>> Its bad file locks on windows. The only suggestion we have is > reboot. > >> >>>> > >> >>>> For some cygwin issues - you need to run 'rebaseall' but this > doesn;t apper to be > >> >>>> that issue. > >> >>>> > >> >>>> If you still have issues - send the complete log [and logfile] to > petsc-maint. > >> >>>> > >> >>>> Satish > >> >>>> > >> >>>>> Thank you all > >> >>>>> > >> >>>>> Fabio > >> >>>>> > >> >> > >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 10 08:31:14 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Aug 2012 08:31:14 -0500 Subject: [petsc-users] Using PetscBarrier or MPI_barrier in Fortran 90 In-Reply-To: <50250543.8080306@york.ac.uk> References: <50250543.8080306@york.ac.uk> Message-ID: On Fri, Aug 10, 2012 at 7:57 AM, Koki Imada wrote: > Hello, > > I'm fairly new to PETSc/MPI programming, and I'm stuck with the use of > barriers. I have a code of the following form: > > 0. Initialise PETSc... > 1. Processor 1 (or the "root") reports to user (using "print" statement). > 2. "Root" does some work, while all other processors do nothing. > 3. "Root" allocates array "A". > 4. All processors does some work on array "A". > 5. "Root" deallocates array "A". > 6. PETSc is finalised. > > In order for this to work, I would like to put barriers between steps 3&4, > and steps 4&5, and also between steps 2&3 (optional). > > Simply putting "call MPI_barrier(MPI_comm_world,**ierr)" between the > steps doesn't do anything and I know this doesn't work due to the lack of > communication between the processors (or otherwise), but I have never been > able to find the "correct" way to implement the barrier... > > If somebody could show me how to correctly implement the barrier (either > MPI or PETSc) between the steps as described above, I will be most grateful. > You really need to read http://www.mcs.anl.gov/research/projects/mpi/usingmpi/. Different processes do not have access to the same memory. Matt > Best Regards, > > Koki > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Aug 10 09:28:48 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Aug 2012 09:28:48 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: On Fri, 10 Aug 2012, Fabio wrote: > I updated to the last version of Petsc then I used again > > PETSC_DIR=`pwd`; export PETSC_DIR > > then in the petsc directory > ./configure --with-cc='win32fe cl' --with-fc=0 > --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > \(x86\)/MPICH2 > > Now I have an error, > "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" > > The directory anyway is there and cygwin could easily access it > > Also trying with mpi-include/lib gives the same error. > What can I do? Remove the --with-mpi-dir option. Configure should automatically look for mpich at this location. Satish From fabio.pinna.2006 at gmail.com Fri Aug 10 10:07:36 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 17:07:36 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: Without using the wiht-mpi-dir option the code start testing but it stops saying the "C compiler associated with mpi-cc win32fe cl does not work" (configure.log is its log) If I use a link to another directory without space in the name I get the same error (provided that the link has been done in cygwin otherwise I have Nonexistant directory). It seems than anyway path are an issue in general (although for standard path they seem to find correctly what they need) The respective log file is configure2.log It look like it is not finding kernel32.lib. Actually a search in my system find only kernel32.dll so I think that the configure script complain rightfully. Anyway could this be linked to my setup win 7 64 bit and MSVC Express (32 bit with some work to do to make it see 64 bit by means of sdk installation)? I thought I could just start with a 32 bit version as it is the default setting of MSVC and it was quite hard to make the 64 bit work (I would have liked to have something simple to focus only on my real implementation, instead of running after VC Express tantrums) Thank you very much for your assistance Fabio 2012/8/10 Satish Balay : > On Fri, 10 Aug 2012, Fabio wrote: > >> I updated to the last version of Petsc then I used again >> >> PETSC_DIR=`pwd`; export PETSC_DIR >> >> then in the petsc directory >> ./configure --with-cc='win32fe cl' --with-fc=0 >> --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ >> \(x86\)/MPICH2 >> >> Now I have an error, >> "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" >> >> The directory anyway is there and cygwin could easily access it >> >> Also trying with mpi-include/lib gives the same error. >> What can I do? > > > Remove the --with-mpi-dir option. > > Configure should automatically look for mpich at this location. > > Satish -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 50495 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure2.log Type: application/octet-stream Size: 50545 bytes Desc: not available URL: From balay at mcs.anl.gov Fri Aug 10 10:25:26 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Aug 2012 10:25:26 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: >>> Executing: /cygdrive/d/Workspace/PETSc/petsc-3.3-p2/bin/win32fe/win32fe cl -o /tmp/petsc-_yEt5c/config.setCompilers/conftest.exe /tmp/petsc-_yEt5c/config.setCompilers/conftest.o sh: LINK : fatal error LNK1104: cannot open file 'kernel32.lib'^M <<< Ok configure failing with compiler check. It hasn't gone up to mpi check. Have you setup compilers in bash correctly [as the installation instructions indicate?]. Have you checked if the compiler works [in both 'compiler cmd' as well as bash] with a simple code - as the instructions indicate? > MSVC Express (32 bit with some work to do to make it see 64 bit by > means of sdk installation)? I don't know what you mean by this. Using 32bit compilers on 64bit windows shouldn't be a problem. You shouldn't need 'some work' to have useable 32bit compilers. Satish On Fri, 10 Aug 2012, Fabio wrote: > Without using the wiht-mpi-dir option the code start testing but it > stops saying the "C compiler associated with mpi-cc win32fe cl does > not work" > (configure.log is its log) > > If I use a link to another directory without space in the name I get > the same error (provided that the link has been done in cygwin > otherwise I have Nonexistant directory). It seems than anyway path are > an issue in general (although for standard path they seem to find > correctly what they need) > The respective log file is configure2.log > > It look like it is not finding kernel32.lib. Actually a search in my > system find only kernel32.dll so I think that the configure script > complain rightfully. > Anyway could this be linked to my setup win 7 64 bit and MSVC Express > (32 bit with some work to do to make it see 64 bit by means of sdk > installation)? > I thought I could just start with a 32 bit version as it is the > default setting of MSVC and it was quite hard to make the 64 bit work > (I would have liked to have something simple to focus only on my real > implementation, instead of running after VC Express tantrums) > > Thank you very much for your assistance > > Fabio > > > 2012/8/10 Satish Balay : > > On Fri, 10 Aug 2012, Fabio wrote: > > > >> I updated to the last version of Petsc then I used again > >> > >> PETSC_DIR=`pwd`; export PETSC_DIR > >> > >> then in the petsc directory > >> ./configure --with-cc='win32fe cl' --with-fc=0 > >> --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > >> \(x86\)/MPICH2 > >> > >> Now I have an error, > >> "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" > >> > >> The directory anyway is there and cygwin could easily access it > >> > >> Also trying with mpi-include/lib gives the same error. > >> What can I do? > > > > > > Remove the --with-mpi-dir option. > > > > Configure should automatically look for mpich at this location. > > > > Satish > From fabio.pinna.2006 at gmail.com Fri Aug 10 10:34:38 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 17:34:38 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: 2012/8/10 Satish Balay : >>>> > Executing: /cygdrive/d/Workspace/PETSc/petsc-3.3-p2/bin/win32fe/win32fe cl -o /tmp/petsc-_yEt5c/config.setCompilers/conftest.exe /tmp/petsc-_yEt5c/config.setCompilers/conftest.o > sh: LINK : fatal error LNK1104: cannot open file 'kernel32.lib'^M > <<< > > Ok configure failing with compiler check. It hasn't gone up to mpi check. > > Have you setup compilers in bash correctly [as the installation > instructions indicate?]. Have you checked if the compiler works [in > both 'compiler cmd' as well as bash] with a simple code - as the > instructions indicate? > >> MSVC Express (32 bit with some work to do to make it see 64 bit by >> means of sdk installation)? > > I don't know what you mean by this. > > Using 32bit compilers on 64bit windows shouldn't be a problem. You > shouldn't need 'some work' to have useable 32bit compilers. > > Satish > > On Fri, 10 Aug 2012, Fabio wrote: > >> Without using the wiht-mpi-dir option the code start testing but it >> stops saying the "C compiler associated with mpi-cc win32fe cl does >> not work" >> (configure.log is its log) >> >> If I use a link to another directory without space in the name I get >> the same error (provided that the link has been done in cygwin >> otherwise I have Nonexistant directory). It seems than anyway path are >> an issue in general (although for standard path they seem to find >> correctly what they need) >> The respective log file is configure2.log >> >> It look like it is not finding kernel32.lib. Actually a search in my >> system find only kernel32.dll so I think that the configure script >> complain rightfully. >> Anyway could this be linked to my setup win 7 64 bit and MSVC Express >> (32 bit with some work to do to make it see 64 bit by means of sdk >> installation)? >> I thought I could just start with a 32 bit version as it is the >> default setting of MSVC and it was quite hard to make the 64 bit work >> (I would have liked to have something simple to focus only on my real >> implementation, instead of running after VC Express tantrums) >> >> Thank you very much for your assistance >> >> Fabio >> >> >> 2012/8/10 Satish Balay : >> > On Fri, 10 Aug 2012, Fabio wrote: >> > >> >> I updated to the last version of Petsc then I used again >> >> >> >> PETSC_DIR=`pwd`; export PETSC_DIR >> >> >> >> then in the petsc directory >> >> ./configure --with-cc='win32fe cl' --with-fc=0 >> >> --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ >> >> \(x86\)/MPICH2 >> >> >> >> Now I have an error, >> >> "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" >> >> >> >> The directory anyway is there and cygwin could easily access it >> >> >> >> Also trying with mpi-include/lib gives the same error. >> >> What can I do? >> > >> > >> > Remove the --with-mpi-dir option. >> > >> > Configure should automatically look for mpich at this location. >> > >> > Satish >> > I start from the VC prompt and then I go to cygwin. When I type cl under cygwin I have the message Microsoft (R) 32-bit C/C++Optimizing Compiler Version ... so I believe cl is really working. Is there something more I could do (in the mean whiel I will write a simple program to test the compiler)? Fabio From knepley at gmail.com Fri Aug 10 10:35:59 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Aug 2012 10:35:59 -0500 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: On Fri, Aug 10, 2012 at 10:34 AM, Fabio wrote: > 2012/8/10 Satish Balay : > >>>> > > Executing: /cygdrive/d/Workspace/PETSc/petsc-3.3-p2/bin/win32fe/win32fe > cl -o /tmp/petsc-_yEt5c/config.setCompilers/conftest.exe > /tmp/petsc-_yEt5c/config.setCompilers/conftest.o > > sh: LINK : fatal error LNK1104: cannot open file 'kernel32.lib'^M > > <<< > > > > Ok configure failing with compiler check. It hasn't gone up to mpi check. > > > > Have you setup compilers in bash correctly [as the installation > > instructions indicate?]. Have you checked if the compiler works [in > > both 'compiler cmd' as well as bash] with a simple code - as the > > instructions indicate? > > > >> MSVC Express (32 bit with some work to do to make it see 64 bit by > >> means of sdk installation)? > > > > I don't know what you mean by this. > > > > Using 32bit compilers on 64bit windows shouldn't be a problem. You > > shouldn't need 'some work' to have useable 32bit compilers. > > > > Satish > > > > On Fri, 10 Aug 2012, Fabio wrote: > > > >> Without using the wiht-mpi-dir option the code start testing but it > >> stops saying the "C compiler associated with mpi-cc win32fe cl does > >> not work" > >> (configure.log is its log) > >> > >> If I use a link to another directory without space in the name I get > >> the same error (provided that the link has been done in cygwin > >> otherwise I have Nonexistant directory). It seems than anyway path are > >> an issue in general (although for standard path they seem to find > >> correctly what they need) > >> The respective log file is configure2.log > >> > >> It look like it is not finding kernel32.lib. Actually a search in my > >> system find only kernel32.dll so I think that the configure script > >> complain rightfully. > >> Anyway could this be linked to my setup win 7 64 bit and MSVC Express > >> (32 bit with some work to do to make it see 64 bit by means of sdk > >> installation)? > >> I thought I could just start with a 32 bit version as it is the > >> default setting of MSVC and it was quite hard to make the 64 bit work > >> (I would have liked to have something simple to focus only on my real > >> implementation, instead of running after VC Express tantrums) > >> > >> Thank you very much for your assistance > >> > >> Fabio > >> > >> > >> 2012/8/10 Satish Balay : > >> > On Fri, 10 Aug 2012, Fabio wrote: > >> > > >> >> I updated to the last version of Petsc then I used again > >> >> > >> >> PETSC_DIR=`pwd`; export PETSC_DIR > >> >> > >> >> then in the petsc directory > >> >> ./configure --with-cc='win32fe cl' --with-fc=0 > >> >> --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ > >> >> \(x86\)/MPICH2 > >> >> > >> >> Now I have an error, > >> >> "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" > >> >> > >> >> The directory anyway is there and cygwin could easily access it > >> >> > >> >> Also trying with mpi-include/lib gives the same error. > >> >> What can I do? > >> > > >> > > >> > Remove the --with-mpi-dir option. > >> > > >> > Configure should automatically look for mpich at this location. > >> > > >> > Satish > >> > > > > I start from the VC prompt and then I go to cygwin. When I type cl > under cygwin I have the message > Microsoft (R) 32-bit C/C++Optimizing Compiler Version ... so I believe > cl is really working. > > Is there something more I could do (in the mean whiel I will write a > simple program to test the compiler)? You need to try and compile something. We can execute the compiler, but it cannot find its libraries. Matt > > Fabio > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Aug 10 10:40:47 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Aug 2012 10:40:47 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: On Fri, 10 Aug 2012, Fabio wrote: > 2012/8/10 Satish Balay : > >>>> > > Executing: /cygdrive/d/Workspace/PETSc/petsc-3.3-p2/bin/win32fe/win32fe cl -o /tmp/petsc-_yEt5c/config.setCompilers/conftest.exe /tmp/petsc-_yEt5c/config.setCompilers/conftest.o > > sh: LINK : fatal error LNK1104: cannot open file 'kernel32.lib'^M > > <<< > > > > Ok configure failing with compiler check. It hasn't gone up to mpi check. > > > > Have you setup compilers in bash correctly [as the installation > > instructions indicate?]. Have you checked if the compiler works [in > > both 'compiler cmd' as well as bash] with a simple code - as the > > instructions indicate? > I start from the VC prompt and then I go to cygwin. When I type cl > under cygwin I have the message > Microsoft (R) 32-bit C/C++Optimizing Compiler Version ... so I believe > cl is really working. > > Is there something more I could do (in the mean whiel I will write a > simple program to test the compiler)? Please compile a simple program - both in 'VC prompt' and in 'bash' Satish From fabio.pinna.2006 at gmail.com Fri Aug 10 10:46:40 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Fri, 10 Aug 2012 17:46:40 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: A simple test.cpp complains about kernel32.lib. My bad, I didn't mean to waste your time. Can I add all the linker option in --with-cc='win32fe cl all-options-VC-need' or I should do it some other way? Fabio 2012/8/10 Matthew Knepley : > On Fri, Aug 10, 2012 at 10:34 AM, Fabio wrote: >> >> 2012/8/10 Satish Balay : >> >>>> >> > Executing: /cygdrive/d/Workspace/PETSc/petsc-3.3-p2/bin/win32fe/win32fe >> > cl -o /tmp/petsc-_yEt5c/config.setCompilers/conftest.exe >> > /tmp/petsc-_yEt5c/config.setCompilers/conftest.o >> > sh: LINK : fatal error LNK1104: cannot open file 'kernel32.lib'^M >> > <<< >> > >> > Ok configure failing with compiler check. It hasn't gone up to mpi >> > check. >> > >> > Have you setup compilers in bash correctly [as the installation >> > instructions indicate?]. Have you checked if the compiler works [in >> > both 'compiler cmd' as well as bash] with a simple code - as the >> > instructions indicate? >> > >> >> MSVC Express (32 bit with some work to do to make it see 64 bit by >> >> means of sdk installation)? >> > >> > I don't know what you mean by this. >> > >> > Using 32bit compilers on 64bit windows shouldn't be a problem. You >> > shouldn't need 'some work' to have useable 32bit compilers. >> > >> > Satish >> > >> > On Fri, 10 Aug 2012, Fabio wrote: >> > >> >> Without using the wiht-mpi-dir option the code start testing but it >> >> stops saying the "C compiler associated with mpi-cc win32fe cl does >> >> not work" >> >> (configure.log is its log) >> >> >> >> If I use a link to another directory without space in the name I get >> >> the same error (provided that the link has been done in cygwin >> >> otherwise I have Nonexistant directory). It seems than anyway path are >> >> an issue in general (although for standard path they seem to find >> >> correctly what they need) >> >> The respective log file is configure2.log >> >> >> >> It look like it is not finding kernel32.lib. Actually a search in my >> >> system find only kernel32.dll so I think that the configure script >> >> complain rightfully. >> >> Anyway could this be linked to my setup win 7 64 bit and MSVC Express >> >> (32 bit with some work to do to make it see 64 bit by means of sdk >> >> installation)? >> >> I thought I could just start with a 32 bit version as it is the >> >> default setting of MSVC and it was quite hard to make the 64 bit work >> >> (I would have liked to have something simple to focus only on my real >> >> implementation, instead of running after VC Express tantrums) >> >> >> >> Thank you very much for your assistance >> >> >> >> Fabio >> >> >> >> >> >> 2012/8/10 Satish Balay : >> >> > On Fri, 10 Aug 2012, Fabio wrote: >> >> > >> >> >> I updated to the last version of Petsc then I used again >> >> >> >> >> >> PETSC_DIR=`pwd`; export PETSC_DIR >> >> >> >> >> >> then in the petsc directory >> >> >> ./configure --with-cc='win32fe cl' --with-fc=0 >> >> >> --download-f2cblas-lapack --with-mpi-dir=/cygdrive/c/Program\ Files\ >> >> >> \(x86\)/MPICH2 >> >> >> >> >> >> Now I have an error, >> >> >> "Nonexistent directory /cygdrive/c/Program\ Files\ \(x86\)/MPICH2" >> >> >> >> >> >> The directory anyway is there and cygwin could easily access it >> >> >> >> >> >> Also trying with mpi-include/lib gives the same error. >> >> >> What can I do? >> >> > >> >> > >> >> > Remove the --with-mpi-dir option. >> >> > >> >> > Configure should automatically look for mpich at this location. >> >> > >> >> > Satish >> >> >> > >> >> I start from the VC prompt and then I go to cygwin. When I type cl >> under cygwin I have the message >> Microsoft (R) 32-bit C/C++Optimizing Compiler Version ... so I believe >> cl is really working. >> >> Is there something more I could do (in the mean whiel I will write a >> simple program to test the compiler)? > > > You need to try and compile something. We can execute the compiler, but it > cannot find its libraries. > > Matt > >> >> >> Fabio > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From balay at mcs.anl.gov Fri Aug 10 10:49:40 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Aug 2012 10:49:40 -0500 (CDT) Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: On Fri, 10 Aug 2012, Fabio wrote: > A simple test.cpp complains about kernel32.lib. My bad, I didn't mean > to waste your time. > Can I add all the linker option in --with-cc='win32fe cl > all-options-VC-need' or I should do it some other way? Perhaps you need to reinstall your compiler. Satish From lzj at umich.edu Fri Aug 10 20:22:01 2012 From: lzj at umich.edu (Peter Zhoujie Lu) Date: Fri, 10 Aug 2012 21:22:01 -0400 Subject: [petsc-users] Segmentation Violation Message-ID: Dear all, I have some issue with my code using petsc. The exact same code runs fine on a linux box with ifort. Is that still be a possible memory corruption? I have some issue with getting valgrind working on 10.7. Is there any better way to debug this? Thanks. Here is the error message I got for OSX 10.7 with gcc. Peter 0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] VecCreateMPIWithArray line 312 /Users/lvzhoujie/Work/petsc-3.3-p2/src/vec/vec/impls/mpi/pbvec.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- p.s. I also found /src/ksp/pc/examples/test/ex7.c not working on my computer. Here is the error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatSetValues()! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex7 on a linux-ifo named peter-lu by plu Fri Aug 10 21:06:23 2012 [0]PETSC ERROR: Libraries linked from /Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/lib [0]PETSC ERROR: Configure run at Fri Aug 10 10:44:21 2012 [0]PETSC ERROR: Configure options --with-pic --download-openmpi=1 --with-fortran=1 --with-debugging=1 --with-shared-libraries=1 --download-superlu_dist=yes --with-fortran-interfaces=1 --with-scalar-type=real --download-f-blas-lapack --with-cc=gcc --with-fc=gfortran --with-cxx=c++ --with-petsc-arch=linux-ifort-opt --download-parmetis=yes --download-metis [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues() line 993 in /Users/lvzhoujie/Work/petsc-3.3-p2/src/mat/interface/matrix.c [0]PETSC ERROR: main() line 25 in src/ksp/pc/examples/tests/ex7.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 73. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 10 20:46:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Aug 2012 20:46:23 -0500 Subject: [petsc-users] Segmentation Violation In-Reply-To: References: Message-ID: On Fri, Aug 10, 2012 at 8:22 PM, Peter Zhoujie Lu wrote: > Dear all, > > I have some issue with my code using petsc. The exact same code runs fine > on a linux box with ifort. Is that still be a possible memory corruption? I > have some issue with getting valgrind working on 10.7. Is there any better > way to debug this? Thanks. Here is the error message I got for OSX 10.7 > with gcc. > This is almost certainly you passing in an invalid pointer for the Vec you are creating. Why not just use gdb? Also, everyone here has valgrind running on OSX. Matt > Peter > > 0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecCreateMPIWithArray line 312 > /Users/lvzhoujie/Work/petsc-3.3-p2/src/vec/vec/impls/mpi/pbvec.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > p.s. I also found /src/ksp/pc/examples/test/ex7.c not working on my > computer. Here is the error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on > argument 1 "mat" before MatSetValues()! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex7 on a linux-ifo named peter-lu by plu Fri Aug 10 > 21:06:23 2012 > [0]PETSC ERROR: Libraries linked from > /Users/lvzhoujie/Work/petsc-3.3-p2/linux-ifort-opt/lib > [0]PETSC ERROR: Configure run at Fri Aug 10 10:44:21 2012 > [0]PETSC ERROR: Configure options --with-pic --download-openmpi=1 > --with-fortran=1 --with-debugging=1 --with-shared-libraries=1 > --download-superlu_dist=yes --with-fortran-interfaces=1 > --with-scalar-type=real --download-f-blas-lapack --with-cc=gcc > --with-fc=gfortran --with-cxx=c++ --with-petsc-arch=linux-ifort-opt > --download-parmetis=yes --download-metis > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetValues() line 993 in > /Users/lvzhoujie/Work/petsc-3.3-p2/src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 25 in src/ksp/pc/examples/tests/ex7.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 73. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pinna.2006 at gmail.com Sat Aug 11 14:02:47 2012 From: fabio.pinna.2006 at gmail.com (Fabio) Date: Sat, 11 Aug 2012 21:02:47 +0200 Subject: [petsc-users] Configure error In-Reply-To: References: <70CDD27D-5ED7-492A-8D27-B3988D5AA8E7@mcs.anl.gov> Message-ID: I solve the configure problem, I just post what I did for future reference. I re-installed Win SDK 7.0 and MSVC 2008 Express. I added the variable LIB C:\Program Files\Microsoft SDKs\Windows\v7.0\Lib;C:\Program Files\Microsoft SDKs\Windows\v7.0\Lib\x86 to the system and user environmental variables. I open a command prompt, I go to the directory C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools I run vsvars.bat. After that I am able to compile with cl test.cpp the run the configure I do simply c:\cygwin\bin\bash.exe --login I go to my PETSc directory and I type in PETSC_DIR=`pwd`; export PETSC_DIR ./configure --with-cc='win32fe cl' --with-fc=0 --download-f2cblaslapack The configure run smooth and complete its job. Unfortunately I have a problem with make now but I'll open another thread. Thanky you very much for your assistance Fabio 2012/8/10 Satish Balay : > On Fri, 10 Aug 2012, Fabio wrote: > >> A simple test.cpp complains about kernel32.lib. My bad, I didn't mean >> to waste your time. >> Can I add all the linker option in --with-cc='win32fe cl >> all-options-VC-need' or I should do it some other way? > > Perhaps you need to reinstall your compiler. > > Satish From aeronova.mailing at gmail.com Sat Aug 11 23:24:53 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Sun, 12 Aug 2012 12:24:53 +0800 Subject: [petsc-users] SLEPc configuration error related to PETSC_ARCH Message-ID: Hello all, I'm having a trouble with installing SLEPc with the PETSC_ARCH variable. I installed PETSc on my Linux as follows: export PETSC_DIR=$PWD export PETSC_ARCH=arch-linux-cxx-debug-real export CXX="/usr/bin/mpicxx" export CC="/usr/bin/mpicc" ./configure --prefix=/home/aeronova/Development/local/lib/petsc/3.3-p2 --download-mpich=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 PETSC_ARCH=arch-linux-cxx-debug-real all make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 PETSC_ARCH=arch-linux-cxx-debug-real install make PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 test Then, I tried to install SLEPc as below export SLEPC_DIR=$PWD export CXX="/usr/bin/mpicxx" export CC="/usr/bin/mpicc" export PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 export PETSC_ARCH=arch-linux-cxx-debug-real ./configure --prefix=/home/aeronova/Development/local/lib/slepc/3.3-p1 , which gives an error: Checking environment... ERROR: PETSc is not configured for architecture arch-linux-cxx-debug-real The error is confusing since I configured/installed PETSc with PETSC_ARCH=arch-linux-cxx-debug-real. I'd appreciate if someone could help me with this problem. Regards, K. Lee. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Sun Aug 12 01:39:53 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sun, 12 Aug 2012 08:39:53 +0200 Subject: [petsc-users] SLEPc configuration error related to PETSC_ARCH In-Reply-To: References: Message-ID: <6C63CDC9-3BEA-4895-8CFF-A9F28B2E50FD@dsic.upv.es> El 12/08/2012, a las 06:24, Kyunghoon Lee escribi?: > Hello all, > > I'm having a trouble with installing SLEPc with the PETSC_ARCH variable. I installed PETSc on my Linux as follows: > > export PETSC_DIR=$PWD > export PETSC_ARCH=arch-linux-cxx-debug-real > export CXX="/usr/bin/mpicxx" > export CC="/usr/bin/mpicc" > ./configure --prefix=/home/aeronova/Development/local/lib/petsc/3.3-p2 --download-mpich=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ > make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 PETSC_ARCH=arch-linux-cxx-debug-real all > make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 PETSC_ARCH=arch-linux-cxx-debug-real install > make PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 test > > Then, I tried to install SLEPc as below > > export SLEPC_DIR=$PWD > export CXX="/usr/bin/mpicxx" > export CC="/usr/bin/mpicc" > export PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 > export PETSC_ARCH=arch-linux-cxx-debug-real > ./configure --prefix=/home/aeronova/Development/local/lib/slepc/3.3-p1 > > , which gives an error: > > Checking environment... > ERROR: PETSc is not configured for architecture arch-linux-cxx-debug-real > > The error is confusing since I configured/installed PETSc with PETSC_ARCH=arch-linux-cxx-debug-real. I'd appreciate if someone could help me with this problem. > > Regards, > K. Lee. > You have to unset PETSC_ARCH. Follow the instructions in section 1.2.4 of SLEPc's manual. Jose From karpeev at mcs.anl.gov Sun Aug 12 03:13:27 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Sun, 12 Aug 2012 03:13:27 -0500 Subject: [petsc-users] [Libmesh-users] Convergence rate for petsc virs In-Reply-To: <816042947.168612.1344728214645.JavaMail.root@mailhub028.itcs.purdue.edu> References: <816042947.168612.1344728214645.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: On Sat, Aug 11, 2012 at 6:36 PM, Subramanya Gautam Sadasiva < ssadasiv at purdue.edu> wrote: > Hi, > I was trying the snes solver on the obstacle problem by modifying > miscellaneous ex 7 and I get only linear convergence times.Is this what is > expected from the virs solver? > Yes, in the sense that you can't expect Newton's typical quadratic convergence: you are not using Newton's method in the usual sense, since the linear (and, in a sense, the nonlinear) problem you are solving changes every time the coincidence set changes. Dmitry. > THanks, > Subramanya > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Libmesh-users mailing list > Libmesh-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/libmesh-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Aug 12 03:17:04 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Sun, 12 Aug 2012 16:17:04 +0800 Subject: [petsc-users] SLEPc configuration error related to PETSC_ARCH In-Reply-To: <6C63CDC9-3BEA-4895-8CFF-A9F28B2E50FD@dsic.upv.es> References: <6C63CDC9-3BEA-4895-8CFF-A9F28B2E50FD@dsic.upv.es> Message-ID: Thanks for the tip; now it works fine. K. Lee. On Sun, Aug 12, 2012 at 2:39 PM, Jose E. Roman wrote: > > El 12/08/2012, a las 06:24, Kyunghoon Lee escribi?: > > > Hello all, > > > > I'm having a trouble with installing SLEPc with the PETSC_ARCH variable. > I installed PETSc on my Linux as follows: > > > > export PETSC_DIR=$PWD > > export PETSC_ARCH=arch-linux-cxx-debug-real > > export CXX="/usr/bin/mpicxx" > > export CC="/usr/bin/mpicc" > > ./configure --prefix=/home/aeronova/Development/local/lib/petsc/3.3-p2 > --download-mpich=1 --download-blacs=1 --download-metis=1 > --download-parmetis=1 --download-scalapack=1 --download-mumps=1 > --download-umfpack=1 --with-clanguage=C++ > > make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 > PETSC_ARCH=arch-linux-cxx-debug-real all > > make PETSC_DIR=/home/aeronova/Development/local/share/petsc/3.3-p2 > PETSC_ARCH=arch-linux-cxx-debug-real install > > make PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 test > > > > Then, I tried to install SLEPc as below > > > > export SLEPC_DIR=$PWD > > export CXX="/usr/bin/mpicxx" > > export CC="/usr/bin/mpicc" > > export PETSC_DIR=/home/aeronova/Development/local/lib/petsc/3.3-p2 > > export PETSC_ARCH=arch-linux-cxx-debug-real > > ./configure --prefix=/home/aeronova/Development/local/lib/slepc/3.3-p1 > > > > , which gives an error: > > > > Checking environment... > > ERROR: PETSc is not configured for architecture arch-linux-cxx-debug-real > > > > The error is confusing since I configured/installed PETSc with > PETSC_ARCH=arch-linux-cxx-debug-real. I'd appreciate if someone could help > me with this problem. > > > > Regards, > > K. Lee. > > > > You have to unset PETSC_ARCH. Follow the instructions in section 1.2.4 of > SLEPc's manual. > Jose > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.hoste at ugent.be Sun Aug 12 10:01:19 2012 From: kenneth.hoste at ugent.be (Kenneth Hoste) Date: Sun, 12 Aug 2012 17:01:19 +0200 Subject: [petsc-users] fixing code w.r.t. new block size argument in VecCreateMPIWithArray Message-ID: <5027C53F.60804@ugent.be> Hello PETSc users, I am trying to build the latest stable release of Trilinos (v10.12.2), with PETSc 3.3-p2 as one of the dependencies. This fails, because Trilinos v10.12.2 doesn't yet cope with some of the changes made in PETSc v3.3, in particular the extra block size argument that was added to the VecCreateMPIWithArray function (as reported on http://www.mcs.anl.gov/petsc/documentation/changes/33.html). My question is simple (I think): how should code that is using the VecCreateMPIWithArray function before the block size argument was introduced be changed such that it compiles and works correctly? Frankly, I have no idea what the code does (I'm not an end-user), I'm just trying to get it to build. My gut feeling says that I should just add the block size argument as '1', which seems a reasonable default is no block size was being used before. So, I would change the current code VecCreateMPIWithArray(comm->USR_comm,inlen,PETSC_DECIDE,x2,&petscX); to something like VecCreateMPIWithArray(comm->USR_comm, 1, inlen,PETSC_DECIDE,x2,&petscX); Is that correct, or is there another way to fix this? regards, Kenneth -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Aug 12 12:01:04 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 12 Aug 2012 12:01:04 -0500 Subject: [petsc-users] fixing code w.r.t. new block size argument in VecCreateMPIWithArray In-Reply-To: <5027C53F.60804@ugent.be> References: <5027C53F.60804@ugent.be> Message-ID: On Aug 12, 2012, at 10:01 AM, Kenneth Hoste wrote: > Hello PETSc users, > > I am trying to build the latest stable release of Trilinos (v10.12.2), with PETSc 3.3-p2 as one of the dependencies. > > This fails, because Trilinos v10.12.2 doesn't yet cope with some of the changes made in PETSc v3.3, in particular the extra block size argument that was added to the VecCreateMPIWithArray function (as reported on http://www.mcs.anl.gov/petsc/documentation/changes/33.html). > > My question is simple (I think): how should code that is using the VecCreateMPIWithArray function before the block size argument was introduced be changed such that it compiles and works correctly? > > Frankly, I have no idea what the code does (I'm not an end-user), I'm just trying to get it to build. > > My gut feeling says that I should just add the block size argument as '1', which seems a reasonable default is no block size was being used before. > So, I would change the current code > > VecCreateMPIWithArray(comm->USR_comm,inlen,PETSC_DECIDE,x2,&petscX); > > to something like > > VecCreateMPIWithArray(comm->USR_comm, 1, inlen,PETSC_DECIDE,x2,&petscX); > > > Is that correct, or is there another way to fix this? Yes, start with this and it will hopefully resolve all the problems. Barry > > > > regards, > > Kenneth From aeronova.mailing at gmail.com Sun Aug 12 23:10:34 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 13 Aug 2012 12:10:34 +0800 Subject: [petsc-users] petsc linking error with MatSetType Message-ID: Hi all, I got the following linking error when I was building libmesh: /homehome//aeronova/aeronova/ Development/Developmentlocal//localshare//sharelibmesh//libmesh/liblib//x86_64-unknownx86_64--unknown-linux-linux-gnu_dbg/gnu_dbg/libmesh.so:libmesh.so undefined : referenceundefined referenceto ` to `MatSetType(_p_Mat*, char const*)'MatSetType(_p_Mat*, char const*)' / /home/aeronovahome//Developmentaeronova/Development//locallocal//share/libmeshshare//lib/libmesh/x86_64-unknownlib/-x86_64-unknownlinux--linuxgnu_dbg/-libmesh.so: gnu_dbg/undefined referencelibmesh.so: to ` undefinedEPSSetTolerances(_p_EPS*, double, int)' reference/ home/toaeronova/ `DevelopmentEPSSetTolerances(_p_EPS*, double, int)'/local/ /share/libmeshhome//libaeronova//x86_64Development/-local/unknownshare/libmesh-linux/lib/-x86_64-gnu_dbg/libmesh.sounknown-: linux-gnu_dbgundefined/libmesh.so reference : toundefined reference ` VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, ScatterMode)to' `/VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, ScatterMode)home/'aeronova /Development/home//localaeronova/Development//localshare//libmeshshare//liblibmesh/x86_64/lib/-unknownx86_64-unknown--linuxlinux--gnu_dbg/gnu_dbglibmesh.so: /libmesh.soundefined: undefined reference reference to to`SNESGetLinearSolveIterations(_p_SNES*, int*) ' `SNESGetLinearSolveIterations(_p_SNES*, int*)/home' It looks like I'm missing some libraries in my petsc, but I'm not sure how to resolve the link problem. My petsc configuration is ./configure --download-mpich=1 --download-blacs=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Aug 13 08:17:50 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 13 Aug 2012 15:17:50 +0200 Subject: [petsc-users] MatMatSolve in sequential call Message-ID: <5028FE7E.30601@gfz-potsdam.de> Hello, Trying to call this routine on 1 proc: call MatMatSolve(F,B,solutions,ierr);CHKERRQ(ierr) I get following crash: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /app on a openmpi-i named glic1 by ag Mon Aug 13 15:06:36 2012 [0]PETSC ERROR: Libraries linked from /home/lib/petsc-3.3-p2/openmpi-intel-complex-debug-f-ds/lib [0]PETSC ERROR: Configure run at Wed Jul 18 12:14:30 2012 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ISGetIndices() line 370 in /home/lib/petsc-3.3-p2/src/vec/is/interface/index.c [0]PETSC ERROR: MatMatSolve_SeqAIJ() line 1109 in /home/lib/petsc-3.3-p2/src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: MatMatSolve() line 3122 in /home/lib/petsc-3.3-p2/src/mat/interface/matrix.c The same code works fine on nproc > 1. Any ideas? -- Regards, Alexander From aeronova.mailing at gmail.com Mon Aug 13 09:09:21 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 13 Aug 2012 22:09:21 +0800 Subject: [petsc-users] petsc building error on mac 10.8 Message-ID: Hi all, I got the following error when I try to build petsc on mac os x 10.8 (mountain lion) --- it failed at MPICH: =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== ***** WARNING: CC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: CXX found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== Running configure on MPICH; this may take several minutes =============================================================================== =============================================================================== Running make on MPICH; this may take several minutes =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make; make install on MPICH: Could not execute "cd /Users/aeronova/Development/local/share/petsc/3.3-p2/externalpackages/mpich2-1.4.1p1 && make && make install": Beginning make ..... Make completed Remove /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libmpich.dylib and rerun make install Remove /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libpmpich.dylib and rerun make install libtool: install: warning: remember to run `libtool --finish /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' libtool: install: warning: remember to run `libtool --finish /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' make: *** [install-preamble] Error 1 This error is confusing since there are a bunch of warnings without an error message. My configuration options are ./configure --prefix=/Users/aeronova/Development/local/lib/petsc/3.3-p2-real --download-mpich=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ I'd appreciate it if anyone can help me with this error. Regards, K. Lee. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 13 09:33:39 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 09:33:39 -0500 Subject: [petsc-users] petsc building error on mac 10.8 In-Reply-To: References: Message-ID: <4060E439-534C-4CD9-81D9-4B13F525C3C1@mcs.anl.gov> For these problems you NEED to send the entire configure.log file to petsc-maint at mcs.anl.gov There is no way we can guess the problem with this little information. Barry On Aug 13, 2012, at 9:09 AM, Kyunghoon Lee wrote: > Hi all, > > I got the following error when I try to build petsc on mac os x 10.8 (mountain lion) --- it failed at MPICH: > > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > =============================================================================== > ***** WARNING: CC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== > ***** WARNING: CXX found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== > Running configure on MPICH; this may take several minutes =============================================================================== =============================================================================== > Running make on MPICH; this may take several minutes =============================================================================== ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error running make; make install on MPICH: Could not execute "cd /Users/aeronova/Development/local/share/petsc/3.3-p2/externalpackages/mpich2-1.4.1p1 && make && make install": > Beginning make > ..... > Make completed > Remove /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libmpich.dylib and rerun make install > Remove /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libpmpich.dylib and rerun make install > libtool: install: warning: remember to run `libtool --finish /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' > libtool: install: warning: remember to run `libtool --finish /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' > make: *** [install-preamble] Error 1 > > This error is confusing since there are a bunch of warnings without an error message. My configuration options are > > ./configure --prefix=/Users/aeronova/Development/local/lib/petsc/3.3-p2-real --download-mpich=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ > > I'd appreciate it if anyone can help me with this error. > > Regards, > K. Lee. From balay at mcs.anl.gov Mon Aug 13 09:39:39 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 13 Aug 2012 09:39:39 -0500 (CDT) Subject: [petsc-users] petsc linking error with MatSetType In-Reply-To: References: Message-ID: This report has incomplete info [you listed errors - but not the link command] does 'make test' work for your PETSc build? If so you should check with libmesh folks about libmesh issues. satish On Mon, 13 Aug 2012, Kyunghoon Lee wrote: > Hi all, > > I got the following linking error when I was building libmesh: > > /homehome//aeronova/aeronova/ > Development/Developmentlocal//localshare//sharelibmesh//libmesh/liblib//x86_64-unknownx86_64--unknown-linux-linux-gnu_dbg/gnu_dbg/libmesh.so:libmesh.so > undefined : referenceundefined referenceto ` to `MatSetType(_p_Mat*, char > const*)'MatSetType(_p_Mat*, char const*)' > / > /home/aeronovahome//Developmentaeronova/Development//locallocal//share/libmeshshare//lib/libmesh/x86_64-unknownlib/-x86_64-unknownlinux--linuxgnu_dbg/-libmesh.so: > gnu_dbg/undefined referencelibmesh.so: to ` > undefinedEPSSetTolerances(_p_EPS*, double, int)' > reference/ home/toaeronova/ `DevelopmentEPSSetTolerances(_p_EPS*, double, > int)'/local/ > /share/libmeshhome//libaeronova//x86_64Development/-local/unknownshare/libmesh-linux/lib/-x86_64-gnu_dbg/libmesh.sounknown-: > linux-gnu_dbgundefined/libmesh.so reference : toundefined reference ` > VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, ScatterMode)to' > `/VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, > ScatterMode)home/'aeronova > /Development/home//localaeronova/Development//localshare//libmeshshare//liblibmesh/x86_64/lib/-unknownx86_64-unknown--linuxlinux--gnu_dbg/gnu_dbglibmesh.so: > /libmesh.soundefined: undefined reference reference to > to`SNESGetLinearSolveIterations(_p_SNES*, int*) ' > `SNESGetLinearSolveIterations(_p_SNES*, int*)/home' > > It looks like I'm missing some libraries in my petsc, but I'm not sure how > to resolve the link problem. My petsc configuration is > > ./configure --download-mpich=1 --download-blacs=1 --download-parmetis=1 > --download-scalapack=1 --download-mumps=1 --download-umfpack=1 > --with-clanguage=C++ > From hzhang at mcs.anl.gov Mon Aug 13 09:54:41 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 13 Aug 2012 09:54:41 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <5028FE7E.30601@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> Message-ID: On Mon, Aug 13, 2012 at 8:17 AM, Alexander Grayver wrote: > Hello, > > Trying to call this routine on 1 proc: > > call MatMatSolve(F,B,solutions,**ierr);CHKERRQ(ierr) > > I get following crash: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > This indicates matrix F (factored matrix) is not provided. I tested petsc-3.3/src/mat/examples/tests/ex129.c with MatMatSovle(): ./ex129 -mat_type aij -dof 2 which works fine. Hong > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: /app on a openmpi-i named glic1 by ag Mon Aug 13 15:06:36 > 2012 > [0]PETSC ERROR: Libraries linked from /home/lib/petsc-3.3-p2/** > openmpi-intel-complex-debug-f-**ds/lib > [0]PETSC ERROR: Configure run at Wed Jul 18 12:14:30 2012 > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: ISGetIndices() line 370 in /home/lib/petsc-3.3-p2/src/** > vec/is/interface/index.c > [0]PETSC ERROR: MatMatSolve_SeqAIJ() line 1109 in > /home/lib/petsc-3.3-p2/src/**mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatMatSolve() line 3122 in /home/lib/petsc-3.3-p2/src/** > mat/interface/matrix.c > > The same code works fine on nproc > 1. > > Any ideas? > > -- > Regards, > Alexander > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Aug 13 10:06:20 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 13 Aug 2012 17:06:20 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> Message-ID: <502917EC.7090600@gfz-potsdam.de> On 13.08.2012 16:54, Hong Zhang wrote: > On Mon, Aug 13, 2012 at 8:17 AM, Alexander Grayver > > wrote: > > Hello, > > Trying to call this routine on 1 proc: > > call MatMatSolve(F,B,solutions,ierr);CHKERRQ(ierr) > > I get following crash: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > > This indicates matrix F (factored matrix) is not provided. > > I tested petsc-3.3/src/mat/examples/tests/ex129.c with MatMatSovle(): > ./ex129 -mat_type aij -dof 2 > which works fine. Hong, I get matrix F as: call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) Please note that all checks in MatMatSolve are done before this crash. So matrix F is provided correctly and it is not zero. The reason is that iscol and isrow == 0 here: PetscErrorCode MatMatSolve_SeqAIJ(Mat A,Mat B,Mat X) { Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; IS iscol = a->col,isrow = a->row; ... ierr = ISGetIndices(isrow,&rout);CHKERRQ(ierr); r = rout; <<<< crash Again, same code runs fine when nproc > 1 > Hong > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 > 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: /app on a openmpi-i named glic1 by ag Mon Aug 13 > 15:06:36 2012 > [0]PETSC ERROR: Libraries linked from > /home/lib/petsc-3.3-p2/openmpi-intel-complex-debug-f-ds/lib > [0]PETSC ERROR: Configure run at Wed Jul 18 12:14:30 2012 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ISGetIndices() line 370 in > /home/lib/petsc-3.3-p2/src/vec/is/interface/index.c > [0]PETSC ERROR: MatMatSolve_SeqAIJ() line 1109 in > /home/lib/petsc-3.3-p2/src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatMatSolve() line 3122 in > /home/lib/petsc-3.3-p2/src/mat/interface/matrix.c > > The same code works fine on nproc > 1. > > Any ideas? > > -- > Regards, > Alexander > > -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From amesga1 at tigers.lsu.edu Mon Aug 13 10:14:48 2012 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Mon, 13 Aug 2012 10:14:48 -0500 Subject: [petsc-users] [Libmesh-users] Convergence rate for petsc virs In-Reply-To: References: <816042947.168612.1344728214645.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Dmitry, On Aug 12, 2012, at 3:13 AM, Dmitry Karpeev wrote: > On Sat, Aug 11, 2012 at 6:36 PM, Subramanya Gautam Sadasiva wrote: > Hi, > I was trying the snes solver on the obstacle problem by modifying miscellaneous ex 7 and I get only linear convergence times.Is this what is expected from the virs solver? > Yes, in the sense that you can't expect Newton's typical quadratic convergence: you are not using Newton's method in the usual sense, > since the linear (and, in a sense, the nonlinear) problem you are solving changes every time the coincidence set changes. You see this is really bad for a case like mine where essentially I am solving a quadratic problem with box constraints (Jac and Constraints are constant during each solve). Is there anything we can do to use this structure shouldn't we be able to get a quadratic convergence ? Thanks, Ata > > Dmitry. > THanks, > Subramanya > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Libmesh-users mailing list > Libmesh-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/libmesh-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Aug 13 10:23:16 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 13 Aug 2012 10:23:16 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <502917EC.7090600@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> Message-ID: Alexander : Hmm, isrow is a matrix ordering, likely a bug :-( Can you send us a stand-alone code that reproduces this error? Do you use an external package for MatMatSolve()? Hong On 13.08.2012 16:54, Hong Zhang wrote: > > On Mon, Aug 13, 2012 at 8:17 AM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> Hello, >> >> Trying to call this routine on 1 proc: >> >> call MatMatSolve(F,B,solutions,ierr);CHKERRQ(ierr) >> >> I get following crash: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Null argument, when expecting valid pointer! >> [0]PETSC ERROR: Null Object: Parameter # 1! >> > > This indicates matrix F (factored matrix) is not provided. > > I tested petsc-3.3/src/mat/examples/tests/ex129.c with MatMatSovle(): > ./ex129 -mat_type aij -dof 2 > which works fine. > > > Hong, > > I get matrix F as: > call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) > > Please note that all checks in MatMatSolve are done before this crash. > So matrix F is provided correctly and it is not zero. > > The reason is that iscol and isrow == 0 here: > > PetscErrorCode MatMatSolve_SeqAIJ(Mat A,Mat B,Mat X) > { > Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; > IS iscol = a->col,isrow = a->row; > > ... > ierr = ISGetIndices(isrow,&rout);CHKERRQ(ierr); r = rout; <<<< crash > > Again, same code runs fine when nproc > 1 > > > Hong > > >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: /app on a openmpi-i named glic1 by ag Mon Aug 13 15:06:36 >> 2012 >> [0]PETSC ERROR: Libraries linked from >> /home/lib/petsc-3.3-p2/openmpi-intel-complex-debug-f-ds/lib >> [0]PETSC ERROR: Configure run at Wed Jul 18 12:14:30 2012 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ISGetIndices() line 370 in >> /home/lib/petsc-3.3-p2/src/vec/is/interface/index.c >> [0]PETSC ERROR: MatMatSolve_SeqAIJ() line 1109 in >> /home/lib/petsc-3.3-p2/src/mat/impls/aij/seq/aijfact.c >> [0]PETSC ERROR: MatMatSolve() line 3122 in >> /home/lib/petsc-3.3-p2/src/mat/interface/matrix.c >> >> The same code works fine on nproc > 1. >> >> Any ideas? >> >> -- >> Regards, >> Alexander >> > > > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Aug 13 10:29:06 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 13 Aug 2012 17:29:06 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> Message-ID: <50291D42.2020102@gfz-potsdam.de> On 13.08.2012 17:23, Hong Zhang wrote: > Alexander : > Hmm, isrow is a matrix ordering, likely a bug :-( > Can you send us a stand-alone code that reproduces this error? Hong, I can try to pull it out, but this may be tricky. > Do you use an external package for MatMatSolve()? Yes, MUMPS. I use ksp with preonly. Originally I thought it will be faster to solve multiple problems using MatMatSolve. This turned out to be not true however, but the code is still like that. -- Regards, Alexander From hzhang at mcs.anl.gov Mon Aug 13 10:56:53 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 13 Aug 2012 10:56:53 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <50291D42.2020102@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> Message-ID: Alexander : petsc-mumps interface does not support MatMatSolve() see ~petsc/src/mat/impls/aij/mpi/mumps/mumps.c Likely MatMatSolve_Basic() is called which calls MatSolve() repeatedly, thus slow. In addition, MUMPS only supports centralized rhs. Petsc interface must scatter distributed rhs into a seqential rhs vector for MUMPS. If num of rhs is large, would be slow here as well. The error in your report likely is a bug in petsc. I'm investigating it and will let you know once it is fixed. Hong On 13.08.2012 17:23, Hong Zhang wrote: > >> Alexander : >> Hmm, isrow is a matrix ordering, likely a bug :-( >> Can you send us a stand-alone code that reproduces this error? >> > > Hong, > > I can try to pull it out, but this may be tricky. > > > Do you use an external package for MatMatSolve()? >> > > Yes, MUMPS. > I use ksp with preonly. > > Originally I thought it will be faster to solve multiple problems using > MatMatSolve. > This turned out to be not true however, but the code is still like that. > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 13 12:00:40 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 13 Aug 2012 11:00:40 -0600 Subject: [petsc-users] [Libmesh-users] Convergence rate for petsc virs In-Reply-To: References: <816042947.168612.1344728214645.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: On Mon, Aug 13, 2012 at 9:14 AM, Ataollah Mesgarnejad < amesga1 at tigers.lsu.edu> wrote: > You see this is really bad for a case like mine where essentially I am > solving a quadratic problem with box constraints (Jac and Constraints are > constant during each solve). Is there anything we can do to use this > structure shouldn't we be able to get a quadratic convergence ? Your problem is not smooth, thus you cannot expect quadratic convergence in a general setting. The active set/reduced space methods (SNESVIRS) change in a discrete way as the bound constraints are activated or deactivate, therefore they do not generally converge quadratically until the constraint set has been identified. You can try semismooth Newton methods (SNESVISS) for a method that has a better chance of converging quadratically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Aug 13 12:01:07 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 13 Aug 2012 12:01:07 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> Message-ID: Alexander : I need a simple code from you to reproduce the error for debugging. If mumps' solver is activated, then you should get error (sequential and parallel) [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: MatMatSolve_MUMPS() is not implemented yet! If petsc's direct solver is activated, then you should not be able to run it in parallel. Please check ~petsc/src/mat/examples/tests/ex125.c to see if you used same calling procedure as this example. Note: I fixed a bug in this example and patched petsc-3.3. The update copy of ex125.c is attached for your convenience. Hong On Mon, Aug 13, 2012 at 10:56 AM, Hong Zhang wrote: > Alexander : > petsc-mumps interface does not support MatMatSolve() > see ~petsc/src/mat/impls/aij/mpi/mumps/mumps.c > Likely MatMatSolve_Basic() is called which calls MatSolve() > repeatedly, thus slow. > > In addition, MUMPS > only supports centralized rhs. Petsc interface must scatter > distributed rhs into a seqential rhs vector for MUMPS. > If num of rhs is large, would be slow here as well. > > The error in your report likely is a bug in petsc. I'm investigating it > and will let you know once it is fixed. > > Hong > > On 13.08.2012 17:23, Hong Zhang wrote: >> >>> Alexander : >>> Hmm, isrow is a matrix ordering, likely a bug :-( >>> Can you send us a stand-alone code that reproduces this error? >>> >> >> Hong, >> >> I can try to pull it out, but this may be tricky. >> >> >> Do you use an external package for MatMatSolve()? >>> >> >> Yes, MUMPS. >> I use ksp with preonly. >> >> Originally I thought it will be faster to solve multiple problems using >> MatMatSolve. >> This turned out to be not true however, but the code is still like that. >> >> -- >> Regards, >> Alexander >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex125.c Type: text/x-csrc Size: 7575 bytes Desc: not available URL: From john.fettig at gmail.com Mon Aug 13 14:01:19 2012 From: john.fettig at gmail.com (John Fettig) Date: Mon, 13 Aug 2012 15:01:19 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> Message-ID: Barry, Thank you for answering my question. I have another one for you: it seems the special case of zero pre-smooths is somewhat non-trivial. The best I can do is set the pre-smoother to Richardson with PCNONE and zero as max_its. However, if you aren't careful in setting KSPSetInitialGuessNonzero this can have unexpected results since the generic KSPSolve will clobber your solution before it even tries a convergence criteria (thus ruling out KSPPREONLY). It also does a couple of unnecessary residual calculations. Would it be reasonable to put a zero-iteration special case in KSPSolve so that if you don't want any iterations it doesn't actually do anything (no setup, no preconditioner, no residual, no scaling, etc.)? Thanks, John On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: > > John, > > On Aug 9, 2012, at 9:50 AM, John Fettig wrote: > >> I am a little confused about what Richardson means. If you use >> multiplicative V-cycle multigrid with Richardson KSP (and no >> convergence monitor), it sets the applyrichardson operator to >> PCApplyRichardson_MG, which appears to just run V-cycles until >> convergence. > > Yes, this is correct. > >> As far as I can tell, it doesn't ever update according >> to the documented >> >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >> > In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. > > >> If on the other hand you use full MG, it does update according to the >> above formula. This also happens if you set a convergence monitor. >> >> I can see how multiplicative V-cycle with Richardson is simply using >> multigrid as a solver. What I don't understand is how full MG with >> Richardson is using multigrid as a solver, because it is using the >> update formula above in between cycles.. Shouldn't there be a >> applyrichardson for full multigrid as well? If not, why? > > I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. > > Barry > >> >> Thanks, >> John > From caplanr at predsci.com Mon Aug 13 14:06:17 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Mon, 13 Aug 2012 12:06:17 -0700 Subject: [petsc-users] mat_view_draw resolution Message-ID: Hi, Is there any way to set the output window size or resolution of mat_view_draw from the command line? - Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 13 15:19:02 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 13 Aug 2012 14:19:02 -0600 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> Message-ID: Shorthand for this is -pc_mg_type kaskade. On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: > Barry, > > Thank you for answering my question. I have another one for you: it > seems the special case of zero pre-smooths is somewhat non-trivial. > The best I can do is set the pre-smoother to Richardson with PCNONE > and zero as max_its. However, if you aren't careful in setting > KSPSetInitialGuessNonzero this can have unexpected results since the > generic KSPSolve will clobber your solution before it even tries a > convergence criteria (thus ruling out KSPPREONLY). It also does a > couple of unnecessary residual calculations. Would it be reasonable to > put a zero-iteration special case in KSPSolve so that if you don't > want any iterations it doesn't actually do anything (no setup, no > preconditioner, no residual, no scaling, etc.)? > > Thanks, > John > > On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: > > > > John, > > > > On Aug 9, 2012, at 9:50 AM, John Fettig wrote: > > > >> I am a little confused about what Richardson means. If you use > >> multiplicative V-cycle multigrid with Richardson KSP (and no > >> convergence monitor), it sets the applyrichardson operator to > >> PCApplyRichardson_MG, which appears to just run V-cycles until > >> convergence. > > > > Yes, this is correct. > > > >> As far as I can tell, it doesn't ever update according > >> to the documented > >> > >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) > >> > > In exact arithmetic it is actually "implicitly" doing exactly this > update. It is difficult to see why this is true generally (because B is > rather complicated for multigrid) but if you consider only two levels with > a direct solver on the coarse grid and SSOR as the pre and post smooth you > can write out the formulas and map back and forth between the two forms. > The reason for the PCApplyRichardson_ forms is because they are a bit more > efficient than separating out the action of B and then doing the update as > above. > > > > > >> If on the other hand you use full MG, it does update according to the > >> above formula. This also happens if you set a convergence monitor. > >> > >> I can see how multiplicative V-cycle with Richardson is simply using > >> multigrid as a solver. What I don't understand is how full MG with > >> Richardson is using multigrid as a solver, because it is using the > >> update formula above in between cycles.. Shouldn't there be a > >> applyrichardson for full multigrid as well? If not, why? > > > > I think there could be a applyRichardson for full multigrid but it > would be kind of complicated and would not benefit much because the amount > of work in a full multigrid step is much higher so the savings would be a > much lower percentage than with V cycle. > > > > Barry > > > >> > >> Thanks, > >> John > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 13 15:22:23 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 15:22:23 -0500 Subject: [petsc-users] mat_view_draw resolution In-Reply-To: References: Message-ID: <1344C86F-F9FC-4322-AF0C-E59E729E3172@mcs.anl.gov> From the manual page for PetscViewerDrawOpen() is -geometry so give this a try. Note also that you can create a PetscViewerDrawOpen() directly in your code setting the size in the code the way you like and pass this viewer to MatView(). Barry On Aug 13, 2012, at 2:06 PM, "Ronald M. Caplan" wrote: > Hi, Is there any way to set the output window size or resolution of mat_view_draw from the command line? > > - Ron From bsmith at mcs.anl.gov Mon Aug 13 15:34:44 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 15:34:44 -0500 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> Message-ID: <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> #undef __FUNCT__ #define __FUNCT__ "PCMGKCycle_Private" PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) { PetscErrorCode ierr; PetscInt i,l = mglevels[0]->levels; PetscFunctionBegin; /* restrict the RHS through all levels to coarsest. */ for (i=l-1; i>0; i--){ if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} ierr = MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} } /* work our way up through the levels */ ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); for (i=0; ieventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} ierr = KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} ierr = MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} } if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} ierr = KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} PetscFunctionReturn(0); } On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: > Shorthand for this is -pc_mg_type kaskade. > > On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: > Barry, > > Thank you for answering my question. I have another one for you: it > seems the special case of zero pre-smooths is somewhat non-trivial. > The best I can do is set the pre-smoother to Richardson with PCNONE > and zero as max_its. However, if you aren't careful in setting > KSPSetInitialGuessNonzero this can have unexpected results since the > generic KSPSolve will clobber your solution before it even tries a > convergence criteria (thus ruling out KSPPREONLY). It also does a > couple of unnecessary residual calculations. Would it be reasonable to > put a zero-iteration special case in KSPSolve so that if you don't > want any iterations it doesn't actually do anything (no setup, no > preconditioner, no residual, no scaling, etc.)? > > Thanks, > John > > On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: > > > > John, > > > > On Aug 9, 2012, at 9:50 AM, John Fettig wrote: > > > >> I am a little confused about what Richardson means. If you use > >> multiplicative V-cycle multigrid with Richardson KSP (and no > >> convergence monitor), it sets the applyrichardson operator to > >> PCApplyRichardson_MG, which appears to just run V-cycles until > >> convergence. > > > > Yes, this is correct. > > > >> As far as I can tell, it doesn't ever update according > >> to the documented > >> > >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) > >> > > In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. > > > > > >> If on the other hand you use full MG, it does update according to the > >> above formula. This also happens if you set a convergence monitor. > >> > >> I can see how multiplicative V-cycle with Richardson is simply using > >> multigrid as a solver. What I don't understand is how full MG with > >> Richardson is using multigrid as a solver, because it is using the > >> update formula above in between cycles.. Shouldn't there be a > >> applyrichardson for full multigrid as well? If not, why? > > > > I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. > > > > Barry > > > >> > >> Thanks, > >> John > > > From john.fettig at gmail.com Mon Aug 13 15:39:36 2012 From: john.fettig at gmail.com (John Fettig) Date: Mon, 13 Aug 2012 16:39:36 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> Message-ID: What if you wanted to do a full cycle with no pre-smooths instead of a v-cycle? John On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: > > #undef __FUNCT__ > #define __FUNCT__ "PCMGKCycle_Private" > PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) > { > PetscErrorCode ierr; > PetscInt i,l = mglevels[0]->levels; > > PetscFunctionBegin; > /* restrict the RHS through all levels to coarsest. */ > for (i=l-1; i>0; i--){ > if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); > if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > } > > /* work our way up through the levels */ > ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); > for (i=0; i if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); > if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); > if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > } > if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); > if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > > PetscFunctionReturn(0); > } > > > On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: > >> Shorthand for this is -pc_mg_type kaskade. >> >> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: >> Barry, >> >> Thank you for answering my question. I have another one for you: it >> seems the special case of zero pre-smooths is somewhat non-trivial. >> The best I can do is set the pre-smoother to Richardson with PCNONE >> and zero as max_its. However, if you aren't careful in setting >> KSPSetInitialGuessNonzero this can have unexpected results since the >> generic KSPSolve will clobber your solution before it even tries a >> convergence criteria (thus ruling out KSPPREONLY). It also does a >> couple of unnecessary residual calculations. Would it be reasonable to >> put a zero-iteration special case in KSPSolve so that if you don't >> want any iterations it doesn't actually do anything (no setup, no >> preconditioner, no residual, no scaling, etc.)? >> >> Thanks, >> John >> >> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: >> > >> > John, >> > >> > On Aug 9, 2012, at 9:50 AM, John Fettig wrote: >> > >> >> I am a little confused about what Richardson means. If you use >> >> multiplicative V-cycle multigrid with Richardson KSP (and no >> >> convergence monitor), it sets the applyrichardson operator to >> >> PCApplyRichardson_MG, which appears to just run V-cycles until >> >> convergence. >> > >> > Yes, this is correct. >> > >> >> As far as I can tell, it doesn't ever update according >> >> to the documented >> >> >> >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >> >> >> > In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. >> > >> > >> >> If on the other hand you use full MG, it does update according to the >> >> above formula. This also happens if you set a convergence monitor. >> >> >> >> I can see how multiplicative V-cycle with Richardson is simply using >> >> multigrid as a solver. What I don't understand is how full MG with >> >> Richardson is using multigrid as a solver, because it is using the >> >> update formula above in between cycles.. Shouldn't there be a >> >> applyrichardson for full multigrid as well? If not, why? >> > >> > I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. >> > >> > Barry >> > >> >> >> >> Thanks, >> >> John >> > >> > From john.fettig at gmail.com Mon Aug 13 15:41:28 2012 From: john.fettig at gmail.com (John Fettig) Date: Mon, 13 Aug 2012 16:41:28 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> Message-ID: It appears the documentation for PCMG has this spelled wrong: -pc_mg_type - multiplicative is the default John On Mon, Aug 13, 2012 at 4:19 PM, Jed Brown wrote: > Shorthand for this is -pc_mg_type kaskade. > > > On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: >> >> Barry, >> >> Thank you for answering my question. I have another one for you: it >> seems the special case of zero pre-smooths is somewhat non-trivial. >> The best I can do is set the pre-smoother to Richardson with PCNONE >> and zero as max_its. However, if you aren't careful in setting >> KSPSetInitialGuessNonzero this can have unexpected results since the >> generic KSPSolve will clobber your solution before it even tries a >> convergence criteria (thus ruling out KSPPREONLY). It also does a >> couple of unnecessary residual calculations. Would it be reasonable to >> put a zero-iteration special case in KSPSolve so that if you don't >> want any iterations it doesn't actually do anything (no setup, no >> preconditioner, no residual, no scaling, etc.)? >> >> Thanks, >> John >> >> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: >> > >> > John, >> > >> > On Aug 9, 2012, at 9:50 AM, John Fettig wrote: >> > >> >> I am a little confused about what Richardson means. If you use >> >> multiplicative V-cycle multigrid with Richardson KSP (and no >> >> convergence monitor), it sets the applyrichardson operator to >> >> PCApplyRichardson_MG, which appears to just run V-cycles until >> >> convergence. >> > >> > Yes, this is correct. >> > >> >> As far as I can tell, it doesn't ever update according >> >> to the documented >> >> >> >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >> >> >> > In exact arithmetic it is actually "implicitly" doing exactly this >> > update. It is difficult to see why this is true generally (because B is >> > rather complicated for multigrid) but if you consider only two levels with a >> > direct solver on the coarse grid and SSOR as the pre and post smooth you can >> > write out the formulas and map back and forth between the two forms. The >> > reason for the PCApplyRichardson_ forms is because they are a bit more >> > efficient than separating out the action of B and then doing the update as >> > above. >> > >> > >> >> If on the other hand you use full MG, it does update according to the >> >> above formula. This also happens if you set a convergence monitor. >> >> >> >> I can see how multiplicative V-cycle with Richardson is simply using >> >> multigrid as a solver. What I don't understand is how full MG with >> >> Richardson is using multigrid as a solver, because it is using the >> >> update formula above in between cycles.. Shouldn't there be a >> >> applyrichardson for full multigrid as well? If not, why? >> > >> > I think there could be a applyRichardson for full multigrid but it >> > would be kind of complicated and would not benefit much because the amount >> > of work in a full multigrid step is much higher so the savings would be a >> > much lower percentage than with V cycle. >> > >> > Barry >> > >> >> >> >> Thanks, >> >> John >> > > > From jedbrown at mcs.anl.gov Mon Aug 13 15:44:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 13 Aug 2012 14:44:38 -0600 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> Message-ID: Thanks. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/194943551e3d On Mon, Aug 13, 2012 at 2:41 PM, John Fettig wrote: > It appears the documentation for PCMG has this spelled wrong: > > -pc_mg_type - multiplicative is > the default > > John > > On Mon, Aug 13, 2012 at 4:19 PM, Jed Brown wrote: > > Shorthand for this is -pc_mg_type kaskade. > > > > > > On Mon, Aug 13, 2012 at 1:01 PM, John Fettig > wrote: > >> > >> Barry, > >> > >> Thank you for answering my question. I have another one for you: it > >> seems the special case of zero pre-smooths is somewhat non-trivial. > >> The best I can do is set the pre-smoother to Richardson with PCNONE > >> and zero as max_its. However, if you aren't careful in setting > >> KSPSetInitialGuessNonzero this can have unexpected results since the > >> generic KSPSolve will clobber your solution before it even tries a > >> convergence criteria (thus ruling out KSPPREONLY). It also does a > >> couple of unnecessary residual calculations. Would it be reasonable to > >> put a zero-iteration special case in KSPSolve so that if you don't > >> want any iterations it doesn't actually do anything (no setup, no > >> preconditioner, no residual, no scaling, etc.)? > >> > >> Thanks, > >> John > >> > >> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: > >> > > >> > John, > >> > > >> > On Aug 9, 2012, at 9:50 AM, John Fettig > wrote: > >> > > >> >> I am a little confused about what Richardson means. If you use > >> >> multiplicative V-cycle multigrid with Richardson KSP (and no > >> >> convergence monitor), it sets the applyrichardson operator to > >> >> PCApplyRichardson_MG, which appears to just run V-cycles until > >> >> convergence. > >> > > >> > Yes, this is correct. > >> > > >> >> As far as I can tell, it doesn't ever update according > >> >> to the documented > >> >> > >> >> x^{n+1} = x^{n} + scale*B(b - A x^{n}) > >> >> > >> > In exact arithmetic it is actually "implicitly" doing exactly > this > >> > update. It is difficult to see why this is true generally (because B > is > >> > rather complicated for multigrid) but if you consider only two levels > with a > >> > direct solver on the coarse grid and SSOR as the pre and post smooth > you can > >> > write out the formulas and map back and forth between the two forms. > The > >> > reason for the PCApplyRichardson_ forms is because they are a bit more > >> > efficient than separating out the action of B and then doing the > update as > >> > above. > >> > > >> > > >> >> If on the other hand you use full MG, it does update according to the > >> >> above formula. This also happens if you set a convergence monitor. > >> >> > >> >> I can see how multiplicative V-cycle with Richardson is simply using > >> >> multigrid as a solver. What I don't understand is how full MG with > >> >> Richardson is using multigrid as a solver, because it is using the > >> >> update formula above in between cycles.. Shouldn't there be a > >> >> applyrichardson for full multigrid as well? If not, why? > >> > > >> > I think there could be a applyRichardson for full multigrid but it > >> > would be kind of complicated and would not benefit much because the > amount > >> > of work in a full multigrid step is much higher so the savings would > be a > >> > much lower percentage than with V cycle. > >> > > >> > Barry > >> > > >> >> > >> >> Thanks, > >> >> John > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Mon Aug 13 16:06:37 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Mon, 13 Aug 2012 17:06:37 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> Message-ID: <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> > No, No, No. You do not have to write a DM implementation. > > You just have to define the data layout in a PetscSection and attach it to > the DM with DMSetDefaultSection. > > Matt Does use of the PetscSection mean that it is neccesary to define a DMMesh? In other words is there a way to create the data layout for the physics without having to specify any information about the mesh? Thanks Colin Quoting Matthew Knepley : > On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev wrote: > >> >> >> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe >> wrote: >> >>> Sanjay, thanks for the reply but I am avoiding using blocked format since >>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 active >>> on a particular node. If I use block the equations I run out of memory >>> pretty quickly on my machine but can get to reasonable sized problems with >>> the unblocked format. >>> >>> Matt, sorry I am not getting this, but I am still not sure how the DM >>> interface works. I can see in the function PCFieldSplitSetDefaults that >>> there is an initial call to DMCreateFieldDecomposition and subsequent calls >>> to DMCreateSubDM based on the command line options. What I am missing is >>> how the first call to DMCreateFieldDecomposition is able to figure out >>> which equations belong to which field just from command line info such as >>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>> >> This might change slightly in the near future in petsc-dev to allow one to >> define splits using named fields. >> In any event, there has to be DM support to implement the decompositions >> over a particular mesh/problem over that mesh. >> With DMDA you can essentially get combinations of strided fields in each >> split. DMCOMPOSITE allows you >> to pull out combinations of the subproblems that were put in there to >> begin with. If you have your own mesh, you have to write >> a DM implementation around it to expose the available fields. >> > > No, No, No. You do not have to write a DM implementation. > > You just have to define the data layout in a PetscSection and attach it to > the DM with DMSetDefaultSection. > > Matt > > >> Dmitry. >> >>> >>> Thanks >>> >>> Colin >>> >>> >>> Quoting Matthew Knepley : >>> >>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>> wrote: >>>> >>>> Colin, >>>>> I you block the equations in FEAP, then the restrained BCs are >>>>> 'included' in assembled PETSc matrix (these dofs have rows that are zero >>>>> modulo a value of unity on the diagonal and the restrained value on the >>>>> right-hand side). >>>>> >>>>> >>>> However, this is not necessary with the DM interface. >>>> >>>> Matt >>>> >>>> >>>> -sg >>>>> >>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>> >>>>> From what I can gather from the petsc-dev source it looks like the >>>>>> commands in 4) will then generate the splits using strided blocks. The >>>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>>> assemble and solve the linear problem within its own nonlinear and time >>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>> boundary >>>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>>> equations. So then in general fields can't be extracted from the active >>>>>> equations using strided blocks and I am stuck with generating the index >>>>>> sets defining the splits on my own. Will it still be possible to make >>>>>> use >>>>>> of the new DM functions in this case? >>>>>> >>>>>> FEAP website: >>>>>> http://www.ce.berkeley.edu/****projects/feap/ >>>>>> >>>>>> > >>>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>> cjm2176 at columbia.edu >>>>>>> >>>>>>>> >wrote: >>>>>>>> >>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>> create >>>>>>>> >>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>> original >>>>>>>>> problem? >>>>>>>>> >>>>>>>>> >>>>>>>>> 1) Use petsc-dev >>>>>>>> >>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>> splits >>>>>>>> however you want, but then recursive gets harder >>>>>>>> >>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>> fields >>>>>>>> apart at once >>>>>>>> >>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>> combination of fields, which from the command line is something like >>>>>>>> >>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>> >>>>>>>> >>>>>>>> I should have shown recursive: >>>>>>> >>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>> >>>>>>> will split 2,0 into two blocks. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Colin McAuliffe >>>>>>>>> PhD Candidate >>>>>>>>> Columbia University >>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>> ------------------------------****----------------- >>>>> >>>>> Sanjay Govindjee, PhD, PE >>>>> Professor of Civil Engineering >>>>> Vice Chair for Academic Affairs >>>>> >>>>> 779 Davis Hall >>>>> Structural Engineering, Mechanics and Materials >>>>> Department of Civil Engineering >>>>> University of California >>>>> Berkeley, CA 94720-1710 >>>>> >>>>> Voice: +1 510 642 6060 >>>>> FAX: +1 510 643 5264 >>>>> s_g at berkeley.edu >>>>> http://www.ce.berkeley.edu/~****sanjay< >>>>> http://www.ce.berkeley.edu/~**sanjay >>>>> > >>>>> ------------------------------****----------------- >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From bsmith at mcs.anl.gov Mon Aug 13 17:53:08 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 17:53:08 -0500 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> Message-ID: <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> John, You are right we do not have completely optimized forms of all variants of the multigrid algorithms. You could make a modification to the routine #undef __FUNCT__ #define __FUNCT__ "PCMGMCycle_Private" PetscErrorCode PCMGMCycle_Private(PC pc,PC_MG_Levels **mglevelsin,PCRichardsonConvergedReason *reason) { PC_MG *mg = (PC_MG*)pc->data; PC_MG_Levels *mgc,*mglevels = *mglevelsin; PetscErrorCode ierr; PetscInt cycles = (mglevels->level == 1) ? 1 : (PetscInt) mglevels->cycles; PetscFunctionBegin; if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} ierr = KSPSolve(mglevels->smoothd,mglevels->b,mglevels->x);CHKERRQ(ierr); /* pre-smooth */ if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} if (mglevels->level) { /* not the coarsest grid */ if (mglevels->eventresidual) {ierr = PetscLogEventBegin(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} ierr = (*mglevels->residual)(mglevels->A,mglevels->b,mglevels->x,mglevels->r);CHKERRQ(ierr); if (mglevels->eventresidual) {ierr = PetscLogEventEnd(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} /* if on finest level and have convergence criteria set */ if (mglevels->level == mglevels->levels-1 && mg->ttol && reason) { PetscReal rnorm; ierr = VecNorm(mglevels->r,NORM_2,&rnorm);CHKERRQ(ierr); if (rnorm <= mg->ttol) { if (rnorm < mg->abstol) { *reason = PCRICHARDSON_CONVERGED_ATOL; ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than absolute tolerance %G\n",rnorm,mg->abstol);CHKERRQ(ierr); } else { *reason = PCRICHARDSON_CONVERGED_RTOL; ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than relative tolerance times initial residual norm %G\n",rnorm,mg->ttol);CHKERRQ(ierr); } PetscFunctionReturn(0); } } mgc = *(mglevelsin - 1); if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} ierr = MatRestrict(mglevels->restrct,mglevels->r,mgc->b);CHKERRQ(ierr); if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} ierr = VecSet(mgc->x,0.0);CHKERRQ(ierr); while (cycles--) { ierr = PCMGMCycle_Private(pc,mglevelsin-1,reason);CHKERRQ(ierr); } if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} ierr = MatInterpolateAdd(mglevels->interpolate,mgc->x,mglevels->x,mglevels->x);CHKERRQ(ierr); if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} ierr = KSPSolve(mglevels->smoothu,mglevels->b,mglevels->x);CHKERRQ(ierr); /* post smooth */ if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} } PetscFunctionReturn(0); } to remove the unneeded computations. Barry On Aug 13, 2012, at 3:39 PM, John Fettig wrote: > What if you wanted to do a full cycle with no pre-smooths instead of a v-cycle? > > John > > On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: >> >> #undef __FUNCT__ >> #define __FUNCT__ "PCMGKCycle_Private" >> PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) >> { >> PetscErrorCode ierr; >> PetscInt i,l = mglevels[0]->levels; >> >> PetscFunctionBegin; >> /* restrict the RHS through all levels to coarsest. */ >> for (i=l-1; i>0; i--){ >> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> ierr = MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); >> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> } >> >> /* work our way up through the levels */ >> ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); >> for (i=0; i> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> ierr = KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); >> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> ierr = MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); >> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> } >> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> ierr = KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); >> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> >> PetscFunctionReturn(0); >> } >> >> >> On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: >> >>> Shorthand for this is -pc_mg_type kaskade. >>> >>> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: >>> Barry, >>> >>> Thank you for answering my question. I have another one for you: it >>> seems the special case of zero pre-smooths is somewhat non-trivial. >>> The best I can do is set the pre-smoother to Richardson with PCNONE >>> and zero as max_its. However, if you aren't careful in setting >>> KSPSetInitialGuessNonzero this can have unexpected results since the >>> generic KSPSolve will clobber your solution before it even tries a >>> convergence criteria (thus ruling out KSPPREONLY). It also does a >>> couple of unnecessary residual calculations. Would it be reasonable to >>> put a zero-iteration special case in KSPSolve so that if you don't >>> want any iterations it doesn't actually do anything (no setup, no >>> preconditioner, no residual, no scaling, etc.)? >>> >>> Thanks, >>> John >>> >>> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: >>>> >>>> John, >>>> >>>> On Aug 9, 2012, at 9:50 AM, John Fettig wrote: >>>> >>>>> I am a little confused about what Richardson means. If you use >>>>> multiplicative V-cycle multigrid with Richardson KSP (and no >>>>> convergence monitor), it sets the applyrichardson operator to >>>>> PCApplyRichardson_MG, which appears to just run V-cycles until >>>>> convergence. >>>> >>>> Yes, this is correct. >>>> >>>>> As far as I can tell, it doesn't ever update according >>>>> to the documented >>>>> >>>>> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >>>>> >>>> In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. >>>> >>>> >>>>> If on the other hand you use full MG, it does update according to the >>>>> above formula. This also happens if you set a convergence monitor. >>>>> >>>>> I can see how multiplicative V-cycle with Richardson is simply using >>>>> multigrid as a solver. What I don't understand is how full MG with >>>>> Richardson is using multigrid as a solver, because it is using the >>>>> update formula above in between cycles.. Shouldn't there be a >>>>> applyrichardson for full multigrid as well? If not, why? >>>> >>>> I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. >>>> >>>> Barry >>>> >>>>> >>>>> Thanks, >>>>> John >>>> >>> >> From jedbrown at mcs.anl.gov Mon Aug 13 17:55:12 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 13 Aug 2012 16:55:12 -0600 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> Message-ID: Should we add PCMGRegisterCycle() and/or provide a more flexible way to specify cycle index? On Mon, Aug 13, 2012 at 4:53 PM, Barry Smith wrote: > > John, > > You are right we do not have completely optimized forms of all > variants of the multigrid algorithms. > > You could make a modification to the routine > > #undef __FUNCT__ > #define __FUNCT__ "PCMGMCycle_Private" > PetscErrorCode PCMGMCycle_Private(PC pc,PC_MG_Levels > **mglevelsin,PCRichardsonConvergedReason *reason) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels *mgc,*mglevels = *mglevelsin; > PetscErrorCode ierr; > PetscInt cycles = (mglevels->level == 1) ? 1 : (PetscInt) > mglevels->cycles; > > PetscFunctionBegin; > > if (mglevels->eventsmoothsolve) {ierr = > PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = > KSPSolve(mglevels->smoothd,mglevels->b,mglevels->x);CHKERRQ(ierr); /* > pre-smooth */ > if (mglevels->eventsmoothsolve) {ierr = > PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->level) { /* not the coarsest grid */ > if (mglevels->eventresidual) {ierr = > PetscLogEventBegin(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > ierr = > (*mglevels->residual)(mglevels->A,mglevels->b,mglevels->x,mglevels->r);CHKERRQ(ierr); > if (mglevels->eventresidual) {ierr = > PetscLogEventEnd(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > > /* if on finest level and have convergence criteria set */ > if (mglevels->level == mglevels->levels-1 && mg->ttol && reason) { > PetscReal rnorm; > ierr = VecNorm(mglevels->r,NORM_2,&rnorm);CHKERRQ(ierr); > if (rnorm <= mg->ttol) { > if (rnorm < mg->abstol) { > *reason = PCRICHARDSON_CONVERGED_ATOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm > %G is less than absolute tolerance %G\n",rnorm,mg->abstol);CHKERRQ(ierr); > } else { > *reason = PCRICHARDSON_CONVERGED_RTOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm > %G is less than relative tolerance times initial residual norm > %G\n",rnorm,mg->ttol);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > } > > mgc = *(mglevelsin - 1); > if (mglevels->eventinterprestrict) {ierr = > PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatRestrict(mglevels->restrct,mglevels->r,mgc->b);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = > PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = VecSet(mgc->x,0.0);CHKERRQ(ierr); > while (cycles--) { > ierr = PCMGMCycle_Private(pc,mglevelsin-1,reason);CHKERRQ(ierr); > } > if (mglevels->eventinterprestrict) {ierr = > PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = > MatInterpolateAdd(mglevels->interpolate,mgc->x,mglevels->x,mglevels->x);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = > PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->eventsmoothsolve) {ierr = > PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = > KSPSolve(mglevels->smoothu,mglevels->b,mglevels->x);CHKERRQ(ierr); /* > post smooth */ > if (mglevels->eventsmoothsolve) {ierr = > PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > } > PetscFunctionReturn(0); > } > > to remove the unneeded computations. > > Barry > > > > On Aug 13, 2012, at 3:39 PM, John Fettig wrote: > > > What if you wanted to do a full cycle with no pre-smooths instead of a > v-cycle? > > > > John > > > > On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: > >> > >> #undef __FUNCT__ > >> #define __FUNCT__ "PCMGKCycle_Private" > >> PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) > >> { > >> PetscErrorCode ierr; > >> PetscInt i,l = mglevels[0]->levels; > >> > >> PetscFunctionBegin; > >> /* restrict the RHS through all levels to coarsest. */ > >> for (i=l-1; i>0; i--){ > >> if (mglevels[i]->eventinterprestrict) {ierr = > PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> ierr = > MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); > >> if (mglevels[i]->eventinterprestrict) {ierr = > PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> } > >> > >> /* work our way up through the levels */ > >> ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); > >> for (i=0; i >> if (mglevels[i]->eventsmoothsolve) {ierr = > PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> ierr = > KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); > >> if (mglevels[i]->eventsmoothsolve) {ierr = > PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> if (mglevels[i+1]->eventinterprestrict) {ierr = > PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> ierr = > MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); > >> if (mglevels[i+1]->eventinterprestrict) {ierr = > PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> } > >> if (mglevels[l-1]->eventsmoothsolve) {ierr = > PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> ierr = > KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); > >> if (mglevels[l-1]->eventsmoothsolve) {ierr = > PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> > >> PetscFunctionReturn(0); > >> } > >> > >> > >> On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: > >> > >>> Shorthand for this is -pc_mg_type kaskade. > >>> > >>> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig > wrote: > >>> Barry, > >>> > >>> Thank you for answering my question. I have another one for you: it > >>> seems the special case of zero pre-smooths is somewhat non-trivial. > >>> The best I can do is set the pre-smoother to Richardson with PCNONE > >>> and zero as max_its. However, if you aren't careful in setting > >>> KSPSetInitialGuessNonzero this can have unexpected results since the > >>> generic KSPSolve will clobber your solution before it even tries a > >>> convergence criteria (thus ruling out KSPPREONLY). It also does a > >>> couple of unnecessary residual calculations. Would it be reasonable to > >>> put a zero-iteration special case in KSPSolve so that if you don't > >>> want any iterations it doesn't actually do anything (no setup, no > >>> preconditioner, no residual, no scaling, etc.)? > >>> > >>> Thanks, > >>> John > >>> > >>> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith > wrote: > >>>> > >>>> John, > >>>> > >>>> On Aug 9, 2012, at 9:50 AM, John Fettig > wrote: > >>>> > >>>>> I am a little confused about what Richardson means. If you use > >>>>> multiplicative V-cycle multigrid with Richardson KSP (and no > >>>>> convergence monitor), it sets the applyrichardson operator to > >>>>> PCApplyRichardson_MG, which appears to just run V-cycles until > >>>>> convergence. > >>>> > >>>> Yes, this is correct. > >>>> > >>>>> As far as I can tell, it doesn't ever update according > >>>>> to the documented > >>>>> > >>>>> x^{n+1} = x^{n} + scale*B(b - A x^{n}) > >>>>> > >>>> In exact arithmetic it is actually "implicitly" doing exactly > this update. It is difficult to see why this is true generally (because B > is rather complicated for multigrid) but if you consider only two levels > with a direct solver on the coarse grid and SSOR as the pre and post smooth > you can write out the formulas and map back and forth between the two > forms. The reason for the PCApplyRichardson_ forms is because they are a > bit more efficient than separating out the action of B and then doing the > update as above. > >>>> > >>>> > >>>>> If on the other hand you use full MG, it does update according to the > >>>>> above formula. This also happens if you set a convergence monitor. > >>>>> > >>>>> I can see how multiplicative V-cycle with Richardson is simply using > >>>>> multigrid as a solver. What I don't understand is how full MG with > >>>>> Richardson is using multigrid as a solver, because it is using the > >>>>> update formula above in between cycles.. Shouldn't there be a > >>>>> applyrichardson for full multigrid as well? If not, why? > >>>> > >>>> I think there could be a applyRichardson for full multigrid but it > would be kind of complicated and would not benefit much because the amount > of work in a full multigrid step is much higher so the savings would be a > much lower percentage than with V cycle. > >>>> > >>>> Barry > >>>> > >>>>> > >>>>> Thanks, > >>>>> John > >>>> > >>> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 13 17:58:20 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 17:58:20 -0500 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> Message-ID: <29D685E1-EF41-46A6-8135-4D9A2F43352F@mcs.anl.gov> On Aug 13, 2012, at 5:55 PM, Jed Brown wrote: > Should we add PCMGRegisterCycle() and/or provide a more flexible way to specify cycle index? For now I'd like to get a better handle on what types of variants there would be. This can be handled by including them and seeing what patterns come up and then later unifying them. Barry > > On Mon, Aug 13, 2012 at 4:53 PM, Barry Smith wrote: > > John, > > You are right we do not have completely optimized forms of all variants of the multigrid algorithms. > > You could make a modification to the routine > > #undef __FUNCT__ > #define __FUNCT__ "PCMGMCycle_Private" > PetscErrorCode PCMGMCycle_Private(PC pc,PC_MG_Levels **mglevelsin,PCRichardsonConvergedReason *reason) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels *mgc,*mglevels = *mglevelsin; > PetscErrorCode ierr; > PetscInt cycles = (mglevels->level == 1) ? 1 : (PetscInt) mglevels->cycles; > > PetscFunctionBegin; > > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels->smoothd,mglevels->b,mglevels->x);CHKERRQ(ierr); /* pre-smooth */ > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->level) { /* not the coarsest grid */ > if (mglevels->eventresidual) {ierr = PetscLogEventBegin(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > ierr = (*mglevels->residual)(mglevels->A,mglevels->b,mglevels->x,mglevels->r);CHKERRQ(ierr); > if (mglevels->eventresidual) {ierr = PetscLogEventEnd(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > > /* if on finest level and have convergence criteria set */ > if (mglevels->level == mglevels->levels-1 && mg->ttol && reason) { > PetscReal rnorm; > ierr = VecNorm(mglevels->r,NORM_2,&rnorm);CHKERRQ(ierr); > if (rnorm <= mg->ttol) { > if (rnorm < mg->abstol) { > *reason = PCRICHARDSON_CONVERGED_ATOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than absolute tolerance %G\n",rnorm,mg->abstol);CHKERRQ(ierr); > } else { > *reason = PCRICHARDSON_CONVERGED_RTOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than relative tolerance times initial residual norm %G\n",rnorm,mg->ttol);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > } > > mgc = *(mglevelsin - 1); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatRestrict(mglevels->restrct,mglevels->r,mgc->b);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = VecSet(mgc->x,0.0);CHKERRQ(ierr); > while (cycles--) { > ierr = PCMGMCycle_Private(pc,mglevelsin-1,reason);CHKERRQ(ierr); > } > if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatInterpolateAdd(mglevels->interpolate,mgc->x,mglevels->x,mglevels->x);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels->smoothu,mglevels->b,mglevels->x);CHKERRQ(ierr); /* post smooth */ > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > } > PetscFunctionReturn(0); > } > > to remove the unneeded computations. > > Barry > > > > On Aug 13, 2012, at 3:39 PM, John Fettig wrote: > > > What if you wanted to do a full cycle with no pre-smooths instead of a v-cycle? > > > > John > > > > On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: > >> > >> #undef __FUNCT__ > >> #define __FUNCT__ "PCMGKCycle_Private" > >> PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) > >> { > >> PetscErrorCode ierr; > >> PetscInt i,l = mglevels[0]->levels; > >> > >> PetscFunctionBegin; > >> /* restrict the RHS through all levels to coarsest. */ > >> for (i=l-1; i>0; i--){ > >> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> ierr = MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); > >> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> } > >> > >> /* work our way up through the levels */ > >> ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); > >> for (i=0; i >> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> ierr = KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); > >> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> ierr = MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); > >> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > >> } > >> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> ierr = KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); > >> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > >> > >> PetscFunctionReturn(0); > >> } > >> > >> > >> On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: > >> > >>> Shorthand for this is -pc_mg_type kaskade. > >>> > >>> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: > >>> Barry, > >>> > >>> Thank you for answering my question. I have another one for you: it > >>> seems the special case of zero pre-smooths is somewhat non-trivial. > >>> The best I can do is set the pre-smoother to Richardson with PCNONE > >>> and zero as max_its. However, if you aren't careful in setting > >>> KSPSetInitialGuessNonzero this can have unexpected results since the > >>> generic KSPSolve will clobber your solution before it even tries a > >>> convergence criteria (thus ruling out KSPPREONLY). It also does a > >>> couple of unnecessary residual calculations. Would it be reasonable to > >>> put a zero-iteration special case in KSPSolve so that if you don't > >>> want any iterations it doesn't actually do anything (no setup, no > >>> preconditioner, no residual, no scaling, etc.)? > >>> > >>> Thanks, > >>> John > >>> > >>> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: > >>>> > >>>> John, > >>>> > >>>> On Aug 9, 2012, at 9:50 AM, John Fettig wrote: > >>>> > >>>>> I am a little confused about what Richardson means. If you use > >>>>> multiplicative V-cycle multigrid with Richardson KSP (and no > >>>>> convergence monitor), it sets the applyrichardson operator to > >>>>> PCApplyRichardson_MG, which appears to just run V-cycles until > >>>>> convergence. > >>>> > >>>> Yes, this is correct. > >>>> > >>>>> As far as I can tell, it doesn't ever update according > >>>>> to the documented > >>>>> > >>>>> x^{n+1} = x^{n} + scale*B(b - A x^{n}) > >>>>> > >>>> In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. > >>>> > >>>> > >>>>> If on the other hand you use full MG, it does update according to the > >>>>> above formula. This also happens if you set a convergence monitor. > >>>>> > >>>>> I can see how multiplicative V-cycle with Richardson is simply using > >>>>> multigrid as a solver. What I don't understand is how full MG with > >>>>> Richardson is using multigrid as a solver, because it is using the > >>>>> update formula above in between cycles.. Shouldn't there be a > >>>>> applyrichardson for full multigrid as well? If not, why? > >>>> > >>>> I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. > >>>> > >>>> Barry > >>>> > >>>>> > >>>>> Thanks, > >>>>> John > >>>> > >>> > >> > > From jedbrown at mcs.anl.gov Mon Aug 13 18:29:10 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 13 Aug 2012 17:29:10 -0600 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: <29D685E1-EF41-46A6-8135-4D9A2F43352F@mcs.anl.gov> References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> <29D685E1-EF41-46A6-8135-4D9A2F43352F@mcs.anl.gov> Message-ID: On Mon, Aug 13, 2012 at 4:58 PM, Barry Smith wrote: > For now I'd like to get a better handle on what types of variants there > would be. This can be handled by including them and seeing what patterns > come up and then later unifying them. Common variants include over-weighting, using a different discretization for residual and correction, and doing a different correction on the second leg of a W cycle. This paper is a good example of what an exotic FAS MG cycle can look like. http://ftp.cs.technion.ac.il/he/people/irad/online-publications/BY93.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 13 19:02:51 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Aug 2012 19:02:51 -0500 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> <29D685E1-EF41-46A6-8135-4D9A2F43352F@mcs.anl.gov> Message-ID: <9FC0CE1E-FC2C-4B9A-B554-DAA666F27DC0@mcs.anl.gov> On Aug 13, 2012, at 6:29 PM, Jed Brown wrote: > On Mon, Aug 13, 2012 at 4:58 PM, Barry Smith wrote: > For now I'd like to get a better handle on what types of variants there would be. This can be handled by including them and seeing what patterns come up and then later unifying them. > > Common variants include over-weighting, using a different discretization for residual and correction, and doing a different correction on the second leg of a W cycle. > > This paper is a good example of what an exotic FAS MG cycle can look like. > > http://ftp.cs.technion.ac.il/he/people/irad/online-publications/BY93.pdf I meant I'd like to see them coded up. Before unifying them. Barry From john.fettig at gmail.com Mon Aug 13 19:49:40 2012 From: john.fettig at gmail.com (John Fettig) Date: Mon, 13 Aug 2012 20:49:40 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> Message-ID: Barry, Thanks, that is essentially what I have done. I just modified it so that before the call to KSPSolve it checks to make sure that max_it for mglevels->smoothd is greater than zero. However, I still can't think of a scenario where you might want to call KSPSolve with max_it == 0 where you would want it to setup the PC, zero the solution vector, etc., so I still think having KSPSolve exit immediately if max_it==0 might be better. John On Mon, Aug 13, 2012 at 6:53 PM, Barry Smith wrote: > > John, > > You are right we do not have completely optimized forms of all variants of the multigrid algorithms. > > You could make a modification to the routine > > #undef __FUNCT__ > #define __FUNCT__ "PCMGMCycle_Private" > PetscErrorCode PCMGMCycle_Private(PC pc,PC_MG_Levels **mglevelsin,PCRichardsonConvergedReason *reason) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels *mgc,*mglevels = *mglevelsin; > PetscErrorCode ierr; > PetscInt cycles = (mglevels->level == 1) ? 1 : (PetscInt) mglevels->cycles; > > PetscFunctionBegin; > > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels->smoothd,mglevels->b,mglevels->x);CHKERRQ(ierr); /* pre-smooth */ > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->level) { /* not the coarsest grid */ > if (mglevels->eventresidual) {ierr = PetscLogEventBegin(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > ierr = (*mglevels->residual)(mglevels->A,mglevels->b,mglevels->x,mglevels->r);CHKERRQ(ierr); > if (mglevels->eventresidual) {ierr = PetscLogEventEnd(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} > > /* if on finest level and have convergence criteria set */ > if (mglevels->level == mglevels->levels-1 && mg->ttol && reason) { > PetscReal rnorm; > ierr = VecNorm(mglevels->r,NORM_2,&rnorm);CHKERRQ(ierr); > if (rnorm <= mg->ttol) { > if (rnorm < mg->abstol) { > *reason = PCRICHARDSON_CONVERGED_ATOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than absolute tolerance %G\n",rnorm,mg->abstol);CHKERRQ(ierr); > } else { > *reason = PCRICHARDSON_CONVERGED_RTOL; > ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm %G is less than relative tolerance times initial residual norm %G\n",rnorm,mg->ttol);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > } > > mgc = *(mglevelsin - 1); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatRestrict(mglevels->restrct,mglevels->r,mgc->b);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = VecSet(mgc->x,0.0);CHKERRQ(ierr); > while (cycles--) { > ierr = PCMGMCycle_Private(pc,mglevelsin-1,reason);CHKERRQ(ierr); > } > if (mglevels->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > ierr = MatInterpolateAdd(mglevels->interpolate,mgc->x,mglevels->x,mglevels->x);CHKERRQ(ierr); > if (mglevels->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > ierr = KSPSolve(mglevels->smoothu,mglevels->b,mglevels->x);CHKERRQ(ierr); /* post smooth */ > if (mglevels->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} > } > PetscFunctionReturn(0); > } > > to remove the unneeded computations. > > Barry > > > > On Aug 13, 2012, at 3:39 PM, John Fettig wrote: > >> What if you wanted to do a full cycle with no pre-smooths instead of a v-cycle? >> >> John >> >> On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: >>> >>> #undef __FUNCT__ >>> #define __FUNCT__ "PCMGKCycle_Private" >>> PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) >>> { >>> PetscErrorCode ierr; >>> PetscInt i,l = mglevels[0]->levels; >>> >>> PetscFunctionBegin; >>> /* restrict the RHS through all levels to coarsest. */ >>> for (i=l-1; i>0; i--){ >>> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >>> ierr = MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); >>> if (mglevels[i]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >>> } >>> >>> /* work our way up through the levels */ >>> ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); >>> for (i=0; i>> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >>> ierr = KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); >>> if (mglevels[i]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >>> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >>> ierr = MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); >>> if (mglevels[i+1]->eventinterprestrict) {ierr = PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >>> } >>> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >>> ierr = KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); >>> if (mglevels[l-1]->eventsmoothsolve) {ierr = PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >>> >>> PetscFunctionReturn(0); >>> } >>> >>> >>> On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: >>> >>>> Shorthand for this is -pc_mg_type kaskade. >>>> >>>> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig wrote: >>>> Barry, >>>> >>>> Thank you for answering my question. I have another one for you: it >>>> seems the special case of zero pre-smooths is somewhat non-trivial. >>>> The best I can do is set the pre-smoother to Richardson with PCNONE >>>> and zero as max_its. However, if you aren't careful in setting >>>> KSPSetInitialGuessNonzero this can have unexpected results since the >>>> generic KSPSolve will clobber your solution before it even tries a >>>> convergence criteria (thus ruling out KSPPREONLY). It also does a >>>> couple of unnecessary residual calculations. Would it be reasonable to >>>> put a zero-iteration special case in KSPSolve so that if you don't >>>> want any iterations it doesn't actually do anything (no setup, no >>>> preconditioner, no residual, no scaling, etc.)? >>>> >>>> Thanks, >>>> John >>>> >>>> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith wrote: >>>>> >>>>> John, >>>>> >>>>> On Aug 9, 2012, at 9:50 AM, John Fettig wrote: >>>>> >>>>>> I am a little confused about what Richardson means. If you use >>>>>> multiplicative V-cycle multigrid with Richardson KSP (and no >>>>>> convergence monitor), it sets the applyrichardson operator to >>>>>> PCApplyRichardson_MG, which appears to just run V-cycles until >>>>>> convergence. >>>>> >>>>> Yes, this is correct. >>>>> >>>>>> As far as I can tell, it doesn't ever update according >>>>>> to the documented >>>>>> >>>>>> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >>>>>> >>>>> In exact arithmetic it is actually "implicitly" doing exactly this update. It is difficult to see why this is true generally (because B is rather complicated for multigrid) but if you consider only two levels with a direct solver on the coarse grid and SSOR as the pre and post smooth you can write out the formulas and map back and forth between the two forms. The reason for the PCApplyRichardson_ forms is because they are a bit more efficient than separating out the action of B and then doing the update as above. >>>>> >>>>> >>>>>> If on the other hand you use full MG, it does update according to the >>>>>> above formula. This also happens if you set a convergence monitor. >>>>>> >>>>>> I can see how multiplicative V-cycle with Richardson is simply using >>>>>> multigrid as a solver. What I don't understand is how full MG with >>>>>> Richardson is using multigrid as a solver, because it is using the >>>>>> update formula above in between cycles.. Shouldn't there be a >>>>>> applyrichardson for full multigrid as well? If not, why? >>>>> >>>>> I think there could be a applyRichardson for full multigrid but it would be kind of complicated and would not benefit much because the amount of work in a full multigrid step is much higher so the savings would be a much lower percentage than with V cycle. >>>>> >>>>> Barry >>>>> >>>>>> >>>>>> Thanks, >>>>>> John >>>>> >>>> >>> > From john.fettig at gmail.com Mon Aug 13 19:57:48 2012 From: john.fettig at gmail.com (John Fettig) Date: Mon, 13 Aug 2012 20:57:48 -0400 Subject: [petsc-users] Richardson with multiplicative MG vs. full MG In-Reply-To: References: <6C9AD0F1-D58D-407E-B839-E9947E1ECE3D@mcs.anl.gov> <92997F7E-8A0B-47DA-9606-3AA06779198E@mcs.anl.gov> <76C04D5F-A337-4FE2-8EB0-589FAB758477@mcs.anl.gov> Message-ID: I really like the idea of PCMGRegisterCycle(). Thanks, John On Mon, Aug 13, 2012 at 6:55 PM, Jed Brown wrote: > Should we add PCMGRegisterCycle() and/or provide a more flexible way to > specify cycle index? > > On Mon, Aug 13, 2012 at 4:53 PM, Barry Smith wrote: >> >> >> John, >> >> You are right we do not have completely optimized forms of all >> variants of the multigrid algorithms. >> >> You could make a modification to the routine >> >> #undef __FUNCT__ >> #define __FUNCT__ "PCMGMCycle_Private" >> PetscErrorCode PCMGMCycle_Private(PC pc,PC_MG_Levels >> **mglevelsin,PCRichardsonConvergedReason *reason) >> { >> PC_MG *mg = (PC_MG*)pc->data; >> PC_MG_Levels *mgc,*mglevels = *mglevelsin; >> PetscErrorCode ierr; >> PetscInt cycles = (mglevels->level == 1) ? 1 : (PetscInt) >> mglevels->cycles; >> >> PetscFunctionBegin; >> >> if (mglevels->eventsmoothsolve) {ierr = >> PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> ierr = >> KSPSolve(mglevels->smoothd,mglevels->b,mglevels->x);CHKERRQ(ierr); /* >> pre-smooth */ >> if (mglevels->eventsmoothsolve) {ierr = >> PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> if (mglevels->level) { /* not the coarsest grid */ >> if (mglevels->eventresidual) {ierr = >> PetscLogEventBegin(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} >> ierr = >> (*mglevels->residual)(mglevels->A,mglevels->b,mglevels->x,mglevels->r);CHKERRQ(ierr); >> if (mglevels->eventresidual) {ierr = >> PetscLogEventEnd(mglevels->eventresidual,0,0,0,0);CHKERRQ(ierr);} >> >> /* if on finest level and have convergence criteria set */ >> if (mglevels->level == mglevels->levels-1 && mg->ttol && reason) { >> PetscReal rnorm; >> ierr = VecNorm(mglevels->r,NORM_2,&rnorm);CHKERRQ(ierr); >> if (rnorm <= mg->ttol) { >> if (rnorm < mg->abstol) { >> *reason = PCRICHARDSON_CONVERGED_ATOL; >> ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm >> %G is less than absolute tolerance %G\n",rnorm,mg->abstol);CHKERRQ(ierr); >> } else { >> *reason = PCRICHARDSON_CONVERGED_RTOL; >> ierr = PetscInfo2(pc,"Linear solver has converged. Residual norm >> %G is less than relative tolerance times initial residual norm >> %G\n",rnorm,mg->ttol);CHKERRQ(ierr); >> } >> PetscFunctionReturn(0); >> } >> } >> >> mgc = *(mglevelsin - 1); >> if (mglevels->eventinterprestrict) {ierr = >> PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> ierr = >> MatRestrict(mglevels->restrct,mglevels->r,mgc->b);CHKERRQ(ierr); >> if (mglevels->eventinterprestrict) {ierr = >> PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> ierr = VecSet(mgc->x,0.0);CHKERRQ(ierr); >> while (cycles--) { >> ierr = PCMGMCycle_Private(pc,mglevelsin-1,reason);CHKERRQ(ierr); >> } >> if (mglevels->eventinterprestrict) {ierr = >> PetscLogEventBegin(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> ierr = >> MatInterpolateAdd(mglevels->interpolate,mgc->x,mglevels->x,mglevels->x);CHKERRQ(ierr); >> if (mglevels->eventinterprestrict) {ierr = >> PetscLogEventEnd(mglevels->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> if (mglevels->eventsmoothsolve) {ierr = >> PetscLogEventBegin(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> ierr = >> KSPSolve(mglevels->smoothu,mglevels->b,mglevels->x);CHKERRQ(ierr); /* >> post smooth */ >> if (mglevels->eventsmoothsolve) {ierr = >> PetscLogEventEnd(mglevels->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> } >> PetscFunctionReturn(0); >> } >> >> to remove the unneeded computations. >> >> Barry >> >> >> >> On Aug 13, 2012, at 3:39 PM, John Fettig wrote: >> >> > What if you wanted to do a full cycle with no pre-smooths instead of a >> > v-cycle? >> > >> > John >> > >> > On Mon, Aug 13, 2012 at 4:34 PM, Barry Smith wrote: >> >> >> >> #undef __FUNCT__ >> >> #define __FUNCT__ "PCMGKCycle_Private" >> >> PetscErrorCode PCMGKCycle_Private(PC pc,PC_MG_Levels **mglevels) >> >> { >> >> PetscErrorCode ierr; >> >> PetscInt i,l = mglevels[0]->levels; >> >> >> >> PetscFunctionBegin; >> >> /* restrict the RHS through all levels to coarsest. */ >> >> for (i=l-1; i>0; i--){ >> >> if (mglevels[i]->eventinterprestrict) {ierr = >> >> PetscLogEventBegin(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> >> ierr = >> >> MatRestrict(mglevels[i]->restrct,mglevels[i]->b,mglevels[i-1]->b);CHKERRQ(ierr); >> >> if (mglevels[i]->eventinterprestrict) {ierr = >> >> PetscLogEventEnd(mglevels[i]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> >> } >> >> >> >> /* work our way up through the levels */ >> >> ierr = VecSet(mglevels[0]->x,0.0);CHKERRQ(ierr); >> >> for (i=0; i> >> if (mglevels[i]->eventsmoothsolve) {ierr = >> >> PetscLogEventBegin(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> >> ierr = >> >> KSPSolve(mglevels[i]->smoothd,mglevels[i]->b,mglevels[i]->x);CHKERRQ(ierr); >> >> if (mglevels[i]->eventsmoothsolve) {ierr = >> >> PetscLogEventEnd(mglevels[i]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> >> if (mglevels[i+1]->eventinterprestrict) {ierr = >> >> PetscLogEventBegin(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> >> ierr = >> >> MatInterpolate(mglevels[i+1]->interpolate,mglevels[i]->x,mglevels[i+1]->x);CHKERRQ(ierr); >> >> if (mglevels[i+1]->eventinterprestrict) {ierr = >> >> PetscLogEventEnd(mglevels[i+1]->eventinterprestrict,0,0,0,0);CHKERRQ(ierr);} >> >> } >> >> if (mglevels[l-1]->eventsmoothsolve) {ierr = >> >> PetscLogEventBegin(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> >> ierr = >> >> KSPSolve(mglevels[l-1]->smoothd,mglevels[l-1]->b,mglevels[l-1]->x);CHKERRQ(ierr); >> >> if (mglevels[l-1]->eventsmoothsolve) {ierr = >> >> PetscLogEventEnd(mglevels[l-1]->eventsmoothsolve,0,0,0,0);CHKERRQ(ierr);} >> >> >> >> PetscFunctionReturn(0); >> >> } >> >> >> >> >> >> On Aug 13, 2012, at 3:19 PM, Jed Brown wrote: >> >> >> >>> Shorthand for this is -pc_mg_type kaskade. >> >>> >> >>> On Mon, Aug 13, 2012 at 1:01 PM, John Fettig >> >>> wrote: >> >>> Barry, >> >>> >> >>> Thank you for answering my question. I have another one for you: it >> >>> seems the special case of zero pre-smooths is somewhat non-trivial. >> >>> The best I can do is set the pre-smoother to Richardson with PCNONE >> >>> and zero as max_its. However, if you aren't careful in setting >> >>> KSPSetInitialGuessNonzero this can have unexpected results since the >> >>> generic KSPSolve will clobber your solution before it even tries a >> >>> convergence criteria (thus ruling out KSPPREONLY). It also does a >> >>> couple of unnecessary residual calculations. Would it be reasonable to >> >>> put a zero-iteration special case in KSPSolve so that if you don't >> >>> want any iterations it doesn't actually do anything (no setup, no >> >>> preconditioner, no residual, no scaling, etc.)? >> >>> >> >>> Thanks, >> >>> John >> >>> >> >>> On Thu, Aug 9, 2012 at 6:37 PM, Barry Smith >> >>> wrote: >> >>>> >> >>>> John, >> >>>> >> >>>> On Aug 9, 2012, at 9:50 AM, John Fettig >> >>>> wrote: >> >>>> >> >>>>> I am a little confused about what Richardson means. If you use >> >>>>> multiplicative V-cycle multigrid with Richardson KSP (and no >> >>>>> convergence monitor), it sets the applyrichardson operator to >> >>>>> PCApplyRichardson_MG, which appears to just run V-cycles until >> >>>>> convergence. >> >>>> >> >>>> Yes, this is correct. >> >>>> >> >>>>> As far as I can tell, it doesn't ever update according >> >>>>> to the documented >> >>>>> >> >>>>> x^{n+1} = x^{n} + scale*B(b - A x^{n}) >> >>>>> >> >>>> In exact arithmetic it is actually "implicitly" doing exactly >> >>>> this update. It is difficult to see why this is true generally (because B >> >>>> is rather complicated for multigrid) but if you consider only two levels >> >>>> with a direct solver on the coarse grid and SSOR as the pre and post smooth >> >>>> you can write out the formulas and map back and forth between the two forms. >> >>>> The reason for the PCApplyRichardson_ forms is because they are a bit more >> >>>> efficient than separating out the action of B and then doing the update as >> >>>> above. >> >>>> >> >>>> >> >>>>> If on the other hand you use full MG, it does update according to >> >>>>> the >> >>>>> above formula. This also happens if you set a convergence monitor. >> >>>>> >> >>>>> I can see how multiplicative V-cycle with Richardson is simply using >> >>>>> multigrid as a solver. What I don't understand is how full MG with >> >>>>> Richardson is using multigrid as a solver, because it is using the >> >>>>> update formula above in between cycles.. Shouldn't there be a >> >>>>> applyrichardson for full multigrid as well? If not, why? >> >>>> >> >>>> I think there could be a applyRichardson for full multigrid but it >> >>>> would be kind of complicated and would not benefit much because the amount >> >>>> of work in a full multigrid step is much higher so the savings would be a >> >>>> much lower percentage than with V cycle. >> >>>> >> >>>> Barry >> >>>> >> >>>>> >> >>>>> Thanks, >> >>>>> John >> >>>> >> >>> >> >> >> > From aeronova.mailing at gmail.com Mon Aug 13 23:44:33 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Tue, 14 Aug 2012 12:44:33 +0800 Subject: [petsc-users] petsc linking error with MatSetType In-Reply-To: References: Message-ID: Thank you for the reply. It turned out the linker tried to use petsc installed by Ubuntu, not the one I built. After removing the one installed by Ubuntu, the error disappeared. K. Lee. On Mon, Aug 13, 2012 at 10:39 PM, Satish Balay wrote: > This report has incomplete info [you listed errors - but not the link > command] > > does 'make test' work for your PETSc build? If so you should check > with libmesh folks about libmesh issues. > > satish > > On Mon, 13 Aug 2012, Kyunghoon Lee wrote: > > > Hi all, > > > > I got the following linking error when I was building libmesh: > > > > /homehome//aeronova/aeronova/ > > > Development/Developmentlocal//localshare//sharelibmesh//libmesh/liblib//x86_64-unknownx86_64--unknown-linux-linux-gnu_dbg/gnu_dbg/libmesh.so:libmesh.so > > undefined : referenceundefined referenceto ` to `MatSetType(_p_Mat*, > char > > const*)'MatSetType(_p_Mat*, char const*)' > > / > > > /home/aeronovahome//Developmentaeronova/Development//locallocal//share/libmeshshare//lib/libmesh/x86_64-unknownlib/-x86_64-unknownlinux--linuxgnu_dbg/-libmesh.so: > > gnu_dbg/undefined referencelibmesh.so: to ` > > undefinedEPSSetTolerances(_p_EPS*, double, int)' > > reference/ home/toaeronova/ `DevelopmentEPSSetTolerances(_p_EPS*, > double, > > int)'/local/ > > > /share/libmeshhome//libaeronova//x86_64Development/-local/unknownshare/libmesh-linux/lib/-x86_64-gnu_dbg/libmesh.sounknown-: > > linux-gnu_dbgundefined/libmesh.so reference : toundefined reference ` > > VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, > ScatterMode)to' > > `/VecScatterEnd(_p_VecScatter*, _p_Vec*, _p_Vec*, InsertMode, > > ScatterMode)home/'aeronova > > > /Development/home//localaeronova/Development//localshare//libmeshshare//liblibmesh/x86_64/lib/-unknownx86_64-unknown--linuxlinux--gnu_dbg/gnu_dbglibmesh.so: > > /libmesh.soundefined: undefined reference reference to > > to`SNESGetLinearSolveIterations(_p_SNES*, int*) ' > > `SNESGetLinearSolveIterations(_p_SNES*, int*)/home' > > > > It looks like I'm missing some libraries in my petsc, but I'm not sure > how > > to resolve the link problem. My petsc configuration is > > > > ./configure --download-mpich=1 --download-blacs=1 --download-parmetis=1 > > --download-scalapack=1 --download-mumps=1 --download-umfpack=1 > > --with-clanguage=C++ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 14 00:38:58 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Aug 2012 00:38:58 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> Message-ID: On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe wrote: > No, No, No. You do not have to write a DM implementation. >> >> You just have to define the data layout in a PetscSection and attach it to >> the DM with DMSetDefaultSection. >> >> Matt >> > > Does use of the PetscSection mean that it is neccesary to define a DMMesh? > In other > words is there a way to create the data layout for the physics without > having to specify > any information about the mesh? > Yes, just call DMSetDefaultSection(). However, note that you will have everything sized correctly, but handling topology and iteration for the residual calculation would be completely up to you. Matt > Thanks > Colin > > > > Quoting Matthew Knepley : > > On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >> wrote: >> >> >>> >>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe >> >wrote: >>> >>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>> since >>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>> active >>>> on a particular node. If I use block the equations I run out of memory >>>> pretty quickly on my machine but can get to reasonable sized problems >>>> with >>>> the unblocked format. >>>> >>>> Matt, sorry I am not getting this, but I am still not sure how the DM >>>> interface works. I can see in the function PCFieldSplitSetDefaults that >>>> there is an initial call to DMCreateFieldDecomposition and subsequent >>>> calls >>>> to DMCreateSubDM based on the command line options. What I am missing is >>>> how the first call to DMCreateFieldDecomposition is able to figure out >>>> which equations belong to which field just from command line info such >>>> as >>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>>> >>>> This might change slightly in the near future in petsc-dev to allow >>> one to >>> define splits using named fields. >>> In any event, there has to be DM support to implement the decompositions >>> over a particular mesh/problem over that mesh. >>> With DMDA you can essentially get combinations of strided fields in each >>> split. DMCOMPOSITE allows you >>> to pull out combinations of the subproblems that were put in there to >>> begin with. If you have your own mesh, you have to write >>> a DM implementation around it to expose the available fields. >>> >>> >> No, No, No. You do not have to write a DM implementation. >> >> You just have to define the data layout in a PetscSection and attach it to >> the DM with DMSetDefaultSection. >> >> Matt >> >> >> Dmitry. >>> >>> >>>> Thanks >>>> >>>> Colin >>>> >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>> >>>>> wrote: >>>>> >>>>> Colin, >>>>> >>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>> 'included' in assembled PETSc matrix (these dofs have rows that are >>>>>> zero >>>>>> modulo a value of unity on the diagonal and the restrained value on >>>>>> the >>>>>> right-hand side). >>>>>> >>>>>> >>>>>> However, this is not necessary with the DM interface. >>>>> >>>>> Matt >>>>> >>>>> >>>>> -sg >>>>> >>>>>> >>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>> >>>>>> From what I can gather from the petsc-dev source it looks like the >>>>>> >>>>>>> commands in 4) will then generate the splits using strided blocks. >>>>>>> The >>>>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>>>> assemble and solve the linear problem within its own nonlinear and >>>>>>> time >>>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>>> boundary >>>>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>>>> equations. So then in general fields can't be extracted from the >>>>>>> active >>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>> index >>>>>>> sets defining the splits on my own. Will it still be possible to make >>>>>>> use >>>>>>> of the new DM functions in this case? >>>>>>> >>>>>>> FEAP website: >>>>>>> http://www.ce.berkeley.edu/******projects/feap/ >>>>>>> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> Quoting Matthew Knepley : >>>>>>> >>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>>> > >>>>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>> cjm2176 at columbia.edu >>>>>>>> >>>>>>>> >wrote: >>>>>>>>> >>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>>> create >>>>>>>>> >>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>>> original >>>>>>>>>> problem? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1) Use petsc-dev >>>>>>>>>> >>>>>>>>> >>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>>> splits >>>>>>>>> however you want, but then recursive gets harder >>>>>>>>> >>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>> fields >>>>>>>>> apart at once >>>>>>>>> >>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>>> combination of fields, which from the command line is something >>>>>>>>> like >>>>>>>>> >>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>> >>>>>>>>> >>>>>>>>> I should have shown recursive: >>>>>>>>> >>>>>>>> >>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>> >>>>>>>> will split 2,0 into two blocks. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Colin McAuliffe >>>>>>>>>> PhD Candidate >>>>>>>>>> Columbia University >>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>> which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>> ------------------------------******----------------- >>>>>> >>>>>> Sanjay Govindjee, PhD, PE >>>>>> Professor of Civil Engineering >>>>>> Vice Chair for Academic Affairs >>>>>> >>>>>> 779 Davis Hall >>>>>> Structural Engineering, Mechanics and Materials >>>>>> Department of Civil Engineering >>>>>> University of California >>>>>> Berkeley, CA 94720-1710 >>>>>> >>>>>> Voice: +1 510 642 6060 >>>>>> FAX: +1 510 643 5264 >>>>>> s_g at berkeley.edu >>>>>> http://www.ce.berkeley.edu/~******sanjay >>>>>> >>>>>> >< >>>>>> http://www.ce.berkeley.edu/~****sanjay >>>>>> >>>>>> > >>>>>> > >>>>>> ------------------------------******----------------- >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Tue Aug 14 04:40:16 2012 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Tue, 14 Aug 2012 15:10:16 +0530 Subject: [petsc-users] Possible causes for segmentation violation error at EPSSetOperators Message-ID: Hi all ! I am using fortran with petsc-3.2-p7, and slepc-3.2-p5 A code snippet of the place where the error is coming: * * *call EPSCreate(PETSC_COMM_WORLD,eps,ierr)* *call EPSSetOperators(eps,AR,PETSC_NULL,ierr) <-------* *call EPSSetProblemType(eps,EPS_HEP,ierr)* The error message: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] EPSSetOperators line 267 src/eps/interface/setup.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./nmaex on a new-linux named shitij by shitij Tue Aug 14 14:39:24 2012 [0]PETSC ERROR: Libraries linked from /home/shitij/NMA/Lib/petsc-3.2-p7/new-linux12/lib [0]PETSC ERROR: Configure run at Wed Jun 20 10:20:34 2012 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/Compiler/11.0/074/mkl/lib/em64t --with-mpi-dir=/home/shitij/NMA/Lib/mpich2-build [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Now, my program conditionally branches in the beginning and then converges somewhere in the middle, after which this piece of code comes. One branch works fine, but the other doesn't. So the usage of the function itself is correct, but something in the other branch is wrong. I can understand this much. What I do not understand is why the error is coming at EPSSetOperators, and how do I debug this problem? I verified that the MPIAIJ array, AR, itself is not null with MatView, and I already created the EPS context as eps in the previous line, and the other arguments to the function are the same as in the other branch, so they cannot be the problem. The values inside the AR array might be incorrect, but they are there so this error should not come because of that as far as I can understand. Any help will be appreciated. Thank you in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Tue Aug 14 05:17:38 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 14 Aug 2012 12:17:38 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> Message-ID: <502A25C2.2070003@gfz-potsdam.de> On 13.08.2012 19:01, Hong Zhang wrote: > Alexander : > I need a simple code from you to reproduce the error for debugging. Hong, Attached you will find test.c that should reproduce this behavior with petsc-3.3-p2. Just run it with 1 and 2 procs as following: ./mpirun -n 1 test -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package mumps -ksp_view Here is matrix I tested it on (15 MB): https://dl.dropbox.com/u/60982984/A.dat > If mumps' solver is activated, then you should get error (sequential > and parallel) > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: MatMatSolve_MUMPS() is not implemented yet! > > If petsc's direct solver is activated, then you should not be able to > run it in parallel. Please note, that it works correctly for nproc > 1 since 1 year or so! So I don't quite understand why there should be error thrown. I just occasionally decided to test something in sequential version and this thing happened. > > Please check ~petsc/src/mat/examples/tests/ex125.c > to see if you used same calling procedure as this example. > Note: I fixed a bug in this example and patched petsc-3.3. > The update copy of ex125.c is attached for your convenience. > > Hong > -- Regards, Alexander -------------- next part -------------- #include #include static char help = []; #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **args) { Vec x,b; Mat A,B,C,F; KSP ksp; PC pc; PetscRandom rctx; PetscInt M,N; PetscErrorCode ierr; PetscViewer viewer; PetscInitialize(&argc,&args,(char *)0,help); ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"A.dat",FILE_MODE_READ,&viewer);CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatLoad(A,viewer);CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); MatGetSize(A,&N,&M); VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,M,&x); VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,N,&b); MatCreateDense(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,10,PETSC_NULL,&B); MatCreateDense(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,10,PETSC_NULL,&C); PetscRandomCreate(PETSC_COMM_WORLD,&rctx); PetscRandomSetFromOptions(rctx); VecSetRandom(b,rctx); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCFactorGetMatrix(pc,&F);CHKERRQ(ierr); ierr = MatMatSolve(F,B,C);CHKERRQ(ierr); KSPDestroy(&ksp); MatDestroy(&A); MatDestroy(&B); MatDestroy(&C); VecDestroy(&x); VecDestroy(&b); PetscRandomDestroy(&rctx); ierr = PetscFinalize(); return 0; } From aeronova.mailing at gmail.com Tue Aug 14 06:26:30 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Tue, 14 Aug 2012 19:26:30 +0800 Subject: [petsc-users] petsc building error on mac 10.8 In-Reply-To: <4060E439-534C-4CD9-81D9-4B13F525C3C1@mcs.anl.gov> References: <4060E439-534C-4CD9-81D9-4B13F525C3C1@mcs.anl.gov> Message-ID: Thank you for the suggestion; I'll send my log file to petsc-maint at mcs.anl.gov. K. Lee. On Mon, Aug 13, 2012 at 10:33 PM, Barry Smith wrote: > > For these problems you NEED to send the entire configure.log file to > petsc-maint at mcs.anl.gov There is no way we can guess the problem with > this little information. > > Barry > > On Aug 13, 2012, at 9:09 AM, Kyunghoon Lee > wrote: > > > Hi all, > > > > I got the following error when I try to build petsc on mac os x 10.8 > (mountain lion) --- it failed at MPICH: > > > > > =============================================================================== > > Configuring PETSc to compile on your system > > > =============================================================================== > > > =============================================================================== > > ***** WARNING: CC found in enviornment variables - ignoring ****** > > =============================================================================== > > =============================================================================== > > ***** WARNING: CXX found in enviornment variables - ignoring ****** > > =============================================================================== > > =============================================================================== > > Running configure on MPICH; this may take several minutes > > =============================================================================== > > =============================================================================== > > Running make on MPICH; this may take several minutes > > =============================================================================== > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > > ------------------------------------------------------------------------------- > > Error running make; make install on MPICH: Could not execute "cd > /Users/aeronova/Development/local/share/petsc/3.3-p2/externalpackages/mpich2-1.4.1p1 > && make && make install": > > Beginning make > > ..... > > Make completed > > Remove > /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libmpich.dylib > and rerun make install > > Remove > /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib/libpmpich.dylib > and rerun make install > > libtool: install: warning: remember to run `libtool --finish > /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' > > libtool: install: warning: remember to run `libtool --finish > /Users/aeronova/Development/local/share/petsc/3.3-p2/arch-darwin-cxx-debug-real/lib' > > make: *** [install-preamble] Error 1 > > > > This error is confusing since there are a bunch of warnings without an > error message. My configuration options are > > > > ./configure > --prefix=/Users/aeronova/Development/local/lib/petsc/3.3-p2-real > --download-mpich=1 --download-blacs=1 --download-metis=1 > --download-parmetis=1 --download-scalapack=1 --download-mumps=1 > --download-umfpack=1 --with-clanguage=C++ > > > > I'd appreciate it if anyone can help me with this error. > > > > Regards, > > K. Lee. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Aug 14 06:31:37 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 14 Aug 2012 13:31:37 +0200 Subject: [petsc-users] Possible causes for segmentation violation error at EPSSetOperators In-Reply-To: References: Message-ID: <073AD380-F4B0-4A78-84E4-523F1FCA3C65@dsic.upv.es> El 14/08/2012, a las 11:40, Shitij Bhargava escribi?: > Hi all ! > > I am using fortran with petsc-3.2-p7, and slepc-3.2-p5 > > A code snippet of the place where the error is coming: > > call EPSCreate(PETSC_COMM_WORLD,eps,ierr) > call EPSSetOperators(eps,AR,PETSC_NULL,ierr) <------- > call EPSSetProblemType(eps,EPS_HEP,ierr) > > The error message: > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] EPSSetOperators line 267 src/eps/interface/setup.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./nmaex on a new-linux named shitij by shitij Tue Aug 14 14:39:24 2012 > [0]PETSC ERROR: Libraries linked from /home/shitij/NMA/Lib/petsc-3.2-p7/new-linux12/lib > [0]PETSC ERROR: Configure run at Wed Jun 20 10:20:34 2012 > [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/Compiler/11.0/074/mkl/lib/em64t --with-mpi-dir=/home/shitij/NMA/Lib/mpich2-build > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > Now, my program conditionally branches in the beginning and then converges somewhere in the middle, after which this piece of code comes. One branch works fine, but the other doesn't. So the usage of the function itself is correct, but something in the other branch is wrong. I can understand this much. > > What I do not understand is why the error is coming at EPSSetOperators, and how do I debug this problem? > > I verified that the MPIAIJ array, AR, itself is not null with MatView, and I already created the EPS context as eps in the previous line, and the other arguments to the function are the same as in the other branch, so they cannot be the problem. The values inside the AR array might be incorrect, but they are there so this error should not come because of that as far as I can understand. > > Any help will be appreciated. > > Thank you in advance. Use PETSC_NULL_OBJECT as in the examples provided in SLEPc: call EPSSetOperators(eps,A,PETSC_NULL_OBJECT,ierr) Jose From shitij.cse at gmail.com Tue Aug 14 08:10:30 2012 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Tue, 14 Aug 2012 18:40:30 +0530 Subject: [petsc-users] Possible causes for segmentation violation error at EPSSetOperators In-Reply-To: <073AD380-F4B0-4A78-84E4-523F1FCA3C65@dsic.upv.es> References: <073AD380-F4B0-4A78-84E4-523F1FCA3C65@dsic.upv.es> Message-ID: Thank you so much. That fixed it. On 14 August 2012 17:01, Jose E. Roman wrote: > > El 14/08/2012, a las 11:40, Shitij Bhargava escribi?: > > > Hi all ! > > > > I am using fortran with petsc-3.2-p7, and slepc-3.2-p5 > > > > A code snippet of the place where the error is coming: > > > > call EPSCreate(PETSC_COMM_WORLD,eps,ierr) > > call EPSSetOperators(eps,AR,PETSC_NULL,ierr) <------- > > call EPSSetProblemType(eps,EPS_HEP,ierr) > > > > The error message: > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] EPSSetOperators line 267 src/eps/interface/setup.c > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 > 09:30:51 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./nmaex on a new-linux named shitij by shitij Tue Aug 14 > 14:39:24 2012 > > [0]PETSC ERROR: Libraries linked from > /home/shitij/NMA/Lib/petsc-3.2-p7/new-linux12/lib > > [0]PETSC ERROR: Configure run at Wed Jun 20 10:20:34 2012 > > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/Compiler/11.0/074/mkl/lib/em64t > --with-mpi-dir=/home/shitij/NMA/Lib/mpich2-build > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > > > Now, my program conditionally branches in the beginning and then > converges somewhere in the middle, after which this piece of code comes. > One branch works fine, but the other doesn't. So the usage of the function > itself is correct, but something in the other branch is wrong. I can > understand this much. > > > > What I do not understand is why the error is coming at EPSSetOperators, > and how do I debug this problem? > > > > I verified that the MPIAIJ array, AR, itself is not null with MatView, > and I already created the EPS context as eps in the previous line, and the > other arguments to the function are the same as in the other branch, so > they cannot be the problem. The values inside the AR array might be > incorrect, but they are there so this error should not come because of that > as far as I can understand. > > > > Any help will be appreciated. > > > > Thank you in advance. > > Use PETSC_NULL_OBJECT as in the examples provided in SLEPc: > > call EPSSetOperators(eps,A,PETSC_NULL_OBJECT,ierr) > > Jose > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From caplanr at predsci.com Tue Aug 14 16:55:16 2012 From: caplanr at predsci.com (Ronald M. Caplan) Date: Tue, 14 Aug 2012 14:55:16 -0700 Subject: [petsc-users] -mat_view_draw error with after-image Message-ID: Hi, When I configure PETSc without after-image, I can use -mat_view_draw and it works fine. I would like to save the image so I re-configured PETSC with after-image and now I get this: X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 1 (X_CreateWindow) Value in failed request: 0xba Serial number of failed request: 7 Current serial number in output stream: 14 I am on a MAC OS X 10.7.4 Any ideas? - Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 14 17:03:17 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Aug 2012 17:03:17 -0500 Subject: [petsc-users] -mat_view_draw error with after-image In-Reply-To: References: Message-ID: <417CB597-863C-485C-A1D0-17EF2CBD1EF0@mcs.anl.gov> Switching this to petsc-maint at mcs.anl.gov Are you sure that afterimage and PETSc are built with the same X? Do otool -L yourexample send to petsc-maint at mcs.anl.gov Also send configure.log and make.log to petsc-maint at mcs.anl.gov Barry On Aug 14, 2012, at 4:55 PM, "Ronald M. Caplan" wrote: > Hi, > > When I configure PETSc without after-image, I can use -mat_view_draw and it works fine. I would like to save the image so I re-configured PETSC with after-image and now I get this: > > X Error of failed request: BadValue (integer parameter out of range for operation) > Major opcode of failed request: 1 (X_CreateWindow) > Value in failed request: 0xba > Serial number of failed request: 7 > Current serial number in output stream: 14 > > I am on a MAC OS X 10.7.4 > > Any ideas? > > - Ron From jzhong at scsolutions.com Tue Aug 14 17:26:39 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 14 Aug 2012 22:26:39 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() Message-ID: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> Dear PETSc folks, I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is MatGetOrdering(A,?) MatGetFactor(A,?) MatLUFactorSymbolic(F, A,?) MatLUFactorNumeric(F, A,?) MatSolve(F,b,x) 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(?). Seemed to be a segmentation fault with the following error message: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! Does anybody have similar experience on that? Thanks a lot! Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 14 17:33:43 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Aug 2012 17:33:43 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> Message-ID: On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: > Dear PETSc folks,**** > > ** ** > > I have a strange observation on using MatLUFactorNumeric() for dense > matrices at different order N. Here is the situation I have:**** > > ** ** > > **1. **I use ./src/mat/tests/ex137.c as an example to direct PETSc > in selecting superLU-dist and mumps. The calling sequence is **** > > MatGetOrdering(A,?)**** > > MatGetFactor(A,?)**** > > MatLUFactorSymbolic(F, A,?)**** > > *MatLUFactorNumeric*(F, A,?)**** > > MatSolve(F,b,x)**** > > **2. **I have three dense matrices A at three different dimensions: > N=75, 2028 and 21180. **** > > **3. **The calling sequence works for N=75 and 2028. But when > N=21180, the program hanged up when calling *MatLUFactorNumeric*(?). > Seemed to be a segmentation fault with the following error message:**** > > ** ** > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [1]PETSC ERROR: Signal received! > ALWAYS send the entire error message. How can we tell anything from a small snippet? Since you have [1], this was run in parallel, so you need 3rd party packages. But you do not seem to be checking return values. Check them to make sure those packages are installed correctly. Matt > Does anybody have similar experience on that?**** > > ** ** > > Thanks a lot!**** > > ** ** > > Jinquan**** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Aug 14 17:39:09 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 14 Aug 2012 22:39:09 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> Thanks, Matt. 1. Yes, I have checked the returned values from x obtained from MatSolve(F,b,x) The norm error check for x is complete for N=75, 2028. 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [391]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [391]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [391]PETSC ERROR: likely location of problem given in stack below [391]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [391]PETSC ERROR: INSTEAD the line number of the start of the function [391]PETSC ERROR: is given. [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] MatLUFactorNumeric line 2778 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interface/matrix.c [391]PETSC ERROR: --------------------- Error Message ------------------------------------ [391]PETSC ERROR: Signal received! [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for recent updates. [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [391]PETSC ERROR: See docs/index.html for manual pages. [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 2012 [391]PETSC ERROR: Libraries linked from /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx-debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 [391]PETSC ERROR: Configure options --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/librefblas.a --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/libreflapack.a --download-blacs --download-scalapack --with-mpi-dir=/usr/local/mvapich2/1.7-gnu --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic --download-mumps --download-superlu_dist --download-parmetis --download-metis --with-fortran-interfaces[391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [cli_391]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, August 14, 2012 3:34 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong > wrote: Dear PETSc folks, I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is MatGetOrdering(A,...) MatGetFactor(A,...) MatLUFactorSymbolic(F, A,...) MatLUFactorNumeric(F, A,...) MatSolve(F,b,x) 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! ALWAYS send the entire error message. How can we tell anything from a small snippet? Since you have [1], this was run in parallel, so you need 3rd party packages. But you do not seem to be checking return values. Check them to make sure those packages are installed correctly. Matt Does anybody have similar experience on that? Thanks a lot! Jinquan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 14 17:52:27 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Aug 2012 17:52:27 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> Message-ID: On Tue, Aug 14, 2012 at 5:39 PM, Jinquan Zhong wrote: > Thanks, Matt.**** > > ** ** > > **1. **Yes, I have checked the returned values from x obtained from* > *** > > MatSolve(F,b,x)**** > > The norm error check for x is complete for N=75, 2028. > If this is C, I want to see ierr = MatSolev(F, b, x);CHKERRQ(ierr); Matt > **** > > **2. **Good point, Matt. Here is the complete message for Rank > 391. The others are similar to this one.**** > > ** ** > > ** ** > > [391]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range**** > > [391]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger**** > > [391]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors**** > > [391]PETSC ERROR: likely location of problem given in stack below**** > > [391]PETSC ERROR: --------------------- Stack Frames > ------------------------------------**** > > [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available,**** > > [391]PETSC ERROR: INSTEAD the line number of the start of the > function**** > > [391]PETSC ERROR: is given.**** > > [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c > **** > > [391]PETSC ERROR: [391] MatLUFactorNumeric line 2778 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interface/matrix.c > **** > > [391]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [391]PETSC ERROR: Signal received!**** > > [391]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 > 15:42:00 CDT 2012**** > > [391]PETSC ERROR: See docs/changes/index.html for recent updates.**** > > [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting.**** > > [391]PETSC ERROR: See docs/index.html for manual pages.**** > > [391]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL on a > arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 2012**** > > [391]PETSC ERROR: Libraries linked from > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx-debug/lib > **** > > [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012**** > > [391]PETSC ERROR: Configure options > --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/librefblas.a > --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/libreflapack.a > --download-blacs --download-scalapack > --with-mpi-dir=/usr/local/mvapich2/1.7-gnu > --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex > --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic > --download-mumps --download-superlu_dist --download-parmetis > --download-metis --with-fortran-interfaces[391]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [391]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file**** > > [cli_391]: aborting job:**** > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391**** > > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Tuesday, August 14, 2012 3:34 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Strange behavior of MatLUFactorNumeric()**** > > ** ** > > On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong > wrote:**** > > Dear PETSc folks,**** > > **** > > I have a strange observation on using MatLUFactorNumeric() for dense > matrices at different order N. Here is the situation I have:**** > > **** > > 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in > selecting superLU-dist and mumps. The calling sequence is **** > > MatGetOrdering(A,?)**** > > MatGetFactor(A,?)**** > > MatLUFactorSymbolic(F, A,?)**** > > *MatLUFactorNumeric*(F, A,?)**** > > MatSolve(F,b,x)**** > > 2. I have three dense matrices A at three different dimensions: > N=75, 2028 and 21180. **** > > 3. The calling sequence works for N=75 and 2028. But when N=21180, > the program hanged up when calling *MatLUFactorNumeric*(?). Seemed to be > a segmentation fault with the following error message:**** > > **** > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [1]PETSC ERROR: Signal received!**** > > ** ** > > ALWAYS send the entire error message. How can we tell anything from a > small snippet?**** > > ** ** > > Since you have [1], this was run in parallel, so you need 3rd party > packages. But you do**** > > not seem to be checking return values. Check them to make sure those > packages are installed**** > > correctly.**** > > ** ** > > Matt**** > > **** > > Does anybody have similar experience on that?**** > > **** > > Thanks a lot!**** > > **** > > Jinquan**** > > > > **** > > ** ** > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Tue Aug 14 17:53:49 2012 From: bourdin at lsu.edu (Blaise Bourdin) Date: Tue, 14 Aug 2012 17:53:49 -0500 Subject: [petsc-users] LineSearch question Message-ID: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> HI, I am trying to understand if the following behavior is normal / expected: I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. The relevant code is as follow: Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) Call VecCopy(solTemp,tmpvec,ierr) Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) Call VecCopy(tmpvec,soltemp,ierr) Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. Blaise Below is the sample output using cp as the linesearch type in which case I get the expected convergence: Solving time step 1: t= 1.00000E+00 0 SNES Function norm 3.742138023215e+00 Line search terminated: lambda = 1, fnorms = 2.1936e-08 1 SNES Function norm 2.193598339906e-08 SNES Object:(temp_) 1 MPI processes type: ls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=15 total number of function evaluations=3 KSP Object: (temp_) 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (temp_) 1 MPI processes type: icc 0 levels of fill tolerance for zero pivot 2.22045e-14 using Manteuffel shift matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Matrix Object: 1 MPI processes type: seqsbaij rows=104, cols=104 package used to perform factorization: petsc total: nonzeros=381, allocated nonzeros=381 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=104, cols=104 total: nonzeros=658, allocated nonzeros=658 total number of mallocs used during MatSetValues calls =0 not using I-node routines SNESLineSearch Object: (temp_) 1 MPI processes type: cp maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=1 SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 0 SNES Function norm 3.742138023215e+00 Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] Line search terminated: lambda = 2, fnorms = 2.1936e-08 1 SNES Function norm 2.193598717772e-08 SNES Object:(temp_) 1 MPI processes type: ls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=15 total number of function evaluations=3 KSP Object: (temp_) 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (temp_) 1 MPI processes type: icc 0 levels of fill tolerance for zero pivot 2.22045e-14 using Manteuffel shift matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Matrix Object: 1 MPI processes type: seqsbaij rows=104, cols=104 package used to perform factorization: petsc total: nonzeros=381, allocated nonzeros=381 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=104, cols=104 total: nonzeros=658, allocated nonzeros=658 total number of mallocs used during MatSetValues calls =0 not using I-node routines SNESLineSearch Object: (temp_) 1 MPI processes type: cp maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=1 SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 when using the default linesearch (bt), the second SNESSolve fails: Solving time step 1: t= 1.00000E+00 0 SNES Function norm 3.742138023215e+00 Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 1 SNES Function norm 2.193598339906e-08 SNES Object:(temp_) 1 MPI processes type: ls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=15 total number of function evaluations=2 KSP Object: (temp_) 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (temp_) 1 MPI processes type: icc 0 levels of fill tolerance for zero pivot 2.22045e-14 using Manteuffel shift matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Matrix Object: 1 MPI processes type: seqsbaij rows=104, cols=104 package used to perform factorization: petsc total: nonzeros=381, allocated nonzeros=381 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=104, cols=104 total: nonzeros=658, allocated nonzeros=658 total number of mallocs used during MatSetValues calls =0 not using I-node routines SNESLineSearch Object: (temp_) 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 0 SNES Function norm 3.742138023215e+00 Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 1 SNES Function norm 1.871069011453e+00 Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 2 SNES Function norm 1.247379340865e+00 Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 3 SNES Function norm 9.355345056231e-01 Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 4 SNES Function norm 7.484276044882e-01 Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 5 SNES Function norm 6.236896704016e-01 Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 6 SNES Function norm 5.345911460556e-01 Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 7 SNES Function norm 4.677672528597e-01 Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 8 SNES Function norm 4.157931136937e-01 Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 9 SNES Function norm 3.742138023528e-01 Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 10 SNES Function norm 3.401943657960e-01 Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 11 SNES Function norm 3.118448353285e-01 Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 12 SNES Function norm 2.878567710844e-01 Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 13 SNES Function norm 2.672955731592e-01 Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 14 SNES Function norm 2.494758682895e-01 Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 15 SNES Function norm 2.338836265275e-01 Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 16 SNES Function norm 2.201257661485e-01 Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 17 SNES Function norm 2.078965569222e-01 Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 18 SNES Function norm 1.969546328772e-01 Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 19 SNES Function norm 1.871069012364e-01 Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 20 SNES Function norm 1.781970487991e-01 Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 21 SNES Function norm 1.700971829468e-01 Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 22 SNES Function norm 1.627016532554e-01 Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 23 SNES Function norm 1.559224177048e-01 Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 24 SNES Function norm 1.496855209981e-01 Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 25 SNES Function norm 1.439283855764e-01 Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 26 SNES Function norm 1.385977046303e-01 Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 27 SNES Function norm 1.336477866088e-01 Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 28 SNES Function norm 1.290392422439e-01 Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 29 SNES Function norm 1.247379341700e-01 Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 30 SNES Function norm 1.207141298427e-01 Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 31 SNES Function norm 1.169418132858e-01 Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 32 SNES Function norm 1.133981219747e-01 Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 33 SNES Function norm 1.100628830937e-01 Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 34 SNES Function norm 1.069182292915e-01 Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 35 SNES Function norm 1.039482784783e-01 Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 36 SNES Function norm 1.011388655469e-01 Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 37 SNES Function norm 9.847731645400e-02 Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 38 SNES Function norm 9.595225705796e-02 Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 39 SNES Function norm 9.355345063171e-02 Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 40 SNES Function norm 9.127165915308e-02 Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 41 SNES Function norm 8.909852441151e-02 Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 42 SNES Function norm 8.702646570443e-02 Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 43 SNES Function norm 8.504859148402e-02 Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 44 SNES Function norm 8.315862278451e-02 Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 45 SNES Function norm 8.135082663716e-02 Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 46 SNES Function norm 7.961995798542e-02 Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 47 SNES Function norm 7.796120886084e-02 Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 48 SNES Function norm 7.637016378216e-02 Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 49 SNES Function norm 7.484276050661e-02 Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 50 SNES Function norm 7.337525539874e-02 SNES Object:(temp_) 1 MPI processes type: ls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=693 total number of function evaluations=51 KSP Object: (temp_) 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (temp_) 1 MPI processes type: icc 0 levels of fill tolerance for zero pivot 2.22045e-14 using Manteuffel shift matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Matrix Object: 1 MPI processes type: seqsbaij rows=104, cols=104 package used to perform factorization: petsc total: nonzeros=381, allocated nonzeros=381 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=104, cols=104 total: nonzeros=658, allocated nonzeros=658 total number of mallocs used during MatSetValues calls =0 not using I-node routines SNESLineSearch Object: (temp_) 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From bsmith at mcs.anl.gov Tue Aug 14 17:54:52 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Aug 2012 17:54:52 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> Message-ID: <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> Can you run with valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: > Thanks, Matt. > > 1. Yes, I have checked the returned values from x obtained from > MatSolve(F,b,x) > > The norm error check for x is complete for N=75, 2028. > > 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. > > > [391]PETSC ERROR: ------------------------------------------------------------------------ > [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [391]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [391]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [391]PETSC ERROR: likely location of problem given in stack below > [391]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [391]PETSC ERROR: INSTEAD the line number of the start of the function > [391]PETSC ERROR: is given. > [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c > [391]PETSC ERROR: [391] MatLUFactorNumeric line 2778 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interface/matrix.c > [391]PETSC ERROR: --------------------- Error Message ------------------------------------ > [391]PETSC ERROR: Signal received! > [391]PETSC ERROR: ------------------------------------------------------------------------ > [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [391]PETSC ERROR: See docs/changes/index.html for recent updates. > [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [391]PETSC ERROR: See docs/index.html for manual pages. > [391]PETSC ERROR: ------------------------------------------------------------------------ > [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 2012 > [391]PETSC ERROR: Libraries linked from /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx-debug/lib > [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 > [391]PETSC ERROR: Configure options --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/librefblas.a --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/libreflapack.a --download-blacs --download-scalapack --with-mpi-dir=/usr/local/mvapich2/1.7-gnu --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic --download-mumps --download-superlu_dist --download-parmetis --download-metis --with-fortran-interfaces[391]PETSC ERROR: ------------------------------------------------------------------------ > [391]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [cli_391]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Tuesday, August 14, 2012 3:34 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: > Dear PETSc folks, > > I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: > > 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is > > MatGetOrdering(A,?) > > MatGetFactor(A,?) > > MatLUFactorSymbolic(F, A,?) > > MatLUFactorNumeric(F, A,?) > > MatSolve(F,b,x) > > 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. > > 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(?). Seemed to be a segmentation fault with the following error message: > > > > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Signal received! > > ALWAYS send the entire error message. How can we tell anything from a small snippet? > > Since you have [1], this was run in parallel, so you need 3rd party packages. But you do > not seem to be checking return values. Check them to make sure those packages are installed > correctly. > > Matt > > Does anybody have similar experience on that? > > Thanks a lot! > > Jinquan > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jzhong at scsolutions.com Tue Aug 14 17:57:28 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 14 Aug 2012 22:57:28 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA316F273890@exchange.scsolutions.com> Here are the related info for N=2028 ... n 0-the LU numfactorization ... done-the LU numfactorization! 0-the MatSolve ... 0-the residual check ... MatSolve: Norm of error 5.7524e-13, resi 1.95473e-18, LU numfact 0 ... Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, August 14, 2012 3:52 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() On Tue, Aug 14, 2012 at 5:39 PM, Jinquan Zhong > wrote: Thanks, Matt. 1. Yes, I have checked the returned values from x obtained from MatSolve(F,b,x) The norm error check for x is complete for N=75, 2028. If this is C, I want to see ierr = MatSolev(F, b, x);CHKERRQ(ierr); Matt 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [391]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [391]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [391]PETSC ERROR: likely location of problem given in stack below [391]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [391]PETSC ERROR: INSTEAD the line number of the start of the function [391]PETSC ERROR: is given. [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] MatLUFactorNumeric line 2778 /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interface/matrix.c [391]PETSC ERROR: --------------------- Error Message ------------------------------------ [391]PETSC ERROR: Signal received! [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for recent updates. [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [391]PETSC ERROR: See docs/index.html for manual pages. [391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 2012 [391]PETSC ERROR: Libraries linked from /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx-debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 [391]PETSC ERROR: Configure options --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/librefblas.a --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib/libreflapack.a --download-blacs --download-scalapack --with-mpi-dir=/usr/local/mvapich2/1.7-gnu --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic --download-mumps --download-superlu_dist --download-parmetis --download-metis --with-fortran-interfaces[391]PETSC ERROR: ------------------------------------------------------------------------ [391]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [cli_391]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, August 14, 2012 3:34 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong > wrote: Dear PETSc folks, I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is MatGetOrdering(A,...) MatGetFactor(A,...) MatLUFactorSymbolic(F, A,...) MatLUFactorNumeric(F, A,...) MatSolve(F,b,x) 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! ALWAYS send the entire error message. How can we tell anything from a small snippet? Since you have [1], this was run in parallel, so you need 3rd party packages. But you do not seem to be checking return values. Check them to make sure those packages are installed correctly. Matt Does anybody have similar experience on that? Thanks a lot! Jinquan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 14 17:58:11 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Aug 2012 17:58:11 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> Message-ID: Blaise, You can run with -snes_linesearch_monitor -info -ksp_monitor_true_residual -ksp_converged_reason to get much more information about what is happening and why the line search is failing. If that doesn't help you can send the code and we can play with it. Barry On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: > HI, > > I am trying to understand if the following behavior is normal / expected: > > I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. > > In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. > > The relevant code is as follow: > Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) > Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) > Call VecCopy(solTemp,tmpvec,ierr) > > Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) > > Call VecCopy(tmpvec,soltemp,ierr) > Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) > > > Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. > > Blaise > > > > Below is the sample output using cp as the linesearch type in which case I get the expected convergence: > Solving time step 1: t= 1.00000E+00 > 0 SNES Function norm 3.742138023215e+00 > Line search terminated: lambda = 1, fnorms = 2.1936e-08 > 1 SNES Function norm 2.193598339906e-08 > SNES Object:(temp_) 1 MPI processes > type: ls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=15 > total number of function evaluations=3 > KSP Object: (temp_) 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (temp_) 1 MPI processes > type: icc > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using Manteuffel shift > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqsbaij > rows=104, cols=104 > package used to perform factorization: petsc > total: nonzeros=381, allocated nonzeros=381 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=104, cols=104 > total: nonzeros=658, allocated nonzeros=658 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > SNESLineSearch Object: (temp_) 1 MPI processes > type: cp > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=1 > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > 0 SNES Function norm 3.742138023215e+00 > Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] > Line search terminated: lambda = 2, fnorms = 2.1936e-08 > 1 SNES Function norm 2.193598717772e-08 > SNES Object:(temp_) 1 MPI processes > type: ls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=15 > total number of function evaluations=3 > KSP Object: (temp_) 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (temp_) 1 MPI processes > type: icc > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using Manteuffel shift > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqsbaij > rows=104, cols=104 > package used to perform factorization: petsc > total: nonzeros=381, allocated nonzeros=381 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=104, cols=104 > total: nonzeros=658, allocated nonzeros=658 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > SNESLineSearch Object: (temp_) 1 MPI processes > type: cp > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=1 > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > > > when using the default linesearch (bt), the second SNESSolve fails: > > Solving time step 1: t= 1.00000E+00 > 0 SNES Function norm 3.742138023215e+00 > Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 > 1 SNES Function norm 2.193598339906e-08 > SNES Object:(temp_) 1 MPI processes > type: ls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=15 > total number of function evaluations=2 > KSP Object: (temp_) 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (temp_) 1 MPI processes > type: icc > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using Manteuffel shift > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqsbaij > rows=104, cols=104 > package used to perform factorization: petsc > total: nonzeros=381, allocated nonzeros=381 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=104, cols=104 > total: nonzeros=658, allocated nonzeros=658 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > SNESLineSearch Object: (temp_) 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=40 > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > 0 SNES Function norm 3.742138023215e+00 > Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > 1 SNES Function norm 1.871069011453e+00 > Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 > 2 SNES Function norm 1.247379340865e+00 > Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 > 3 SNES Function norm 9.355345056231e-01 > Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 > 4 SNES Function norm 7.484276044882e-01 > Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 > 5 SNES Function norm 6.236896704016e-01 > Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 > 6 SNES Function norm 5.345911460556e-01 > Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 > 7 SNES Function norm 4.677672528597e-01 > Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 > 8 SNES Function norm 4.157931136937e-01 > Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 > 9 SNES Function norm 3.742138023528e-01 > Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 > 10 SNES Function norm 3.401943657960e-01 > Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 > 11 SNES Function norm 3.118448353285e-01 > Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 > 12 SNES Function norm 2.878567710844e-01 > Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 > 13 SNES Function norm 2.672955731592e-01 > Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 > 14 SNES Function norm 2.494758682895e-01 > Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 > 15 SNES Function norm 2.338836265275e-01 > Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 > 16 SNES Function norm 2.201257661485e-01 > Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 > 17 SNES Function norm 2.078965569222e-01 > Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 > 18 SNES Function norm 1.969546328772e-01 > Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 > 19 SNES Function norm 1.871069012364e-01 > Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 > 20 SNES Function norm 1.781970487991e-01 > Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 > 21 SNES Function norm 1.700971829468e-01 > Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 > 22 SNES Function norm 1.627016532554e-01 > Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 > 23 SNES Function norm 1.559224177048e-01 > Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 > 24 SNES Function norm 1.496855209981e-01 > Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 > 25 SNES Function norm 1.439283855764e-01 > Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 > 26 SNES Function norm 1.385977046303e-01 > Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 > 27 SNES Function norm 1.336477866088e-01 > Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 > 28 SNES Function norm 1.290392422439e-01 > Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 > 29 SNES Function norm 1.247379341700e-01 > Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 > 30 SNES Function norm 1.207141298427e-01 > Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 > 31 SNES Function norm 1.169418132858e-01 > Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 > 32 SNES Function norm 1.133981219747e-01 > Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 > 33 SNES Function norm 1.100628830937e-01 > Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 > 34 SNES Function norm 1.069182292915e-01 > Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 > 35 SNES Function norm 1.039482784783e-01 > Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 > 36 SNES Function norm 1.011388655469e-01 > Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 > 37 SNES Function norm 9.847731645400e-02 > Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 > 38 SNES Function norm 9.595225705796e-02 > Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 > 39 SNES Function norm 9.355345063171e-02 > Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 > 40 SNES Function norm 9.127165915308e-02 > Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 > 41 SNES Function norm 8.909852441151e-02 > Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 > 42 SNES Function norm 8.702646570443e-02 > Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 > 43 SNES Function norm 8.504859148402e-02 > Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 > 44 SNES Function norm 8.315862278451e-02 > Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 > 45 SNES Function norm 8.135082663716e-02 > Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 > 46 SNES Function norm 7.961995798542e-02 > Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 > 47 SNES Function norm 7.796120886084e-02 > Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 > 48 SNES Function norm 7.637016378216e-02 > Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 > 49 SNES Function norm 7.484276050661e-02 > Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 > 50 SNES Function norm 7.337525539874e-02 > SNES Object:(temp_) 1 MPI processes > type: ls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=693 > total number of function evaluations=51 > KSP Object: (temp_) 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (temp_) 1 MPI processes > type: icc > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using Manteuffel shift > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqsbaij > rows=104, cols=104 > package used to perform factorization: petsc > total: nonzeros=381, allocated nonzeros=381 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=104, cols=104 > total: nonzeros=658, allocated nonzeros=658 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > SNESLineSearch Object: (temp_) 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=40 > SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 > > > > > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > > From jzhong at scsolutions.com Tue Aug 14 18:05:19 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 14 Aug 2012 23:05:19 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> Message-ID: <7237384C7A8F2642A8854B751619FA316F2738A5@exchange.scsolutions.com> Barry, The machine I ran this program does not have valgrind. Another interesting observation is that when I ran the same three matrices using PETSc3.2. MatLUFactorNumeric() hanged up even on N=75, 2028 till I specified -mat_superlu_dist_colperm. However, MatLUFactorNumeric() didn't work for N=21180 either even I used -mat_superlu_dist_rowperm NATURAL -mat_superlu_dist_colperm NATURAL -mat_superlu_dist_parsymbfact YES I suspect that there is something incompatible in the factored matrix from superLU-dist to be used MatLUFactorNumeric() in PETSc3.2. Although PETSc 3.3 fixed this issue for matrix with small N, however, this issue relapsed for large N in PETSc3.3. Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Tuesday, August 14, 2012 3:55 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() Can you run with valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: > Thanks, Matt. > > 1. Yes, I have checked the returned values from x obtained from > MatSolve(F,b,x) > > The norm error check for x is complete for N=75, 2028. > > 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. > > > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range [391]PETSC ERROR: Try > option -start_in_debugger or -on_error_attach_debugger [391]PETSC > ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors [391]PETSC ERROR: likely location of > problem given in stack below [391]PETSC ERROR: --------------------- > Stack Frames ------------------------------------ > [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [391]PETSC ERROR: INSTEAD the line number of the start of the function > [391]PETSC ERROR: is given. > [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/ai > j/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] > MatLUFactorNumeric line 2778 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interfac > e/matrix.c [391]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [391]PETSC ERROR: Signal received! > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 > 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for > recent updates. > [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [391]PETSC ERROR: See docs/index.html for manual pages. > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL > on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 > 2012 [391]PETSC ERROR: Libraries linked from > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx- > debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 > [391]PETSC ERROR: Configure options > --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib > /librefblas.a > --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/l > ib/libreflapack.a --download-blacs --download-scalapack > --with-mpi-dir=/usr/local/mvapich2/1.7-gnu > --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex > --with-precision=double --with-clanguage=cxx > --with-fortran-kernels=generic --download-mumps > --download-superlu_dist --download-parmetis --download-metis > --with-fortran-interfaces[391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: User provided function() line 0 in unknown > directory unknown file > [cli_391]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 > > > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Tuesday, August 14, 2012 3:34 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: > Dear PETSc folks, > > I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: > > 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is > > MatGetOrdering(A,...) > > MatGetFactor(A,...) > > MatLUFactorSymbolic(F, A,...) > > MatLUFactorNumeric(F, A,...) > > MatSolve(F,b,x) > > 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. > > 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: > > > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Signal received! > > ALWAYS send the entire error message. How can we tell anything from a small snippet? > > Since you have [1], this was run in parallel, so you need 3rd party > packages. But you do not seem to be checking return values. Check them > to make sure those packages are installed correctly. > > Matt > > Does anybody have similar experience on that? > > Thanks a lot! > > Jinquan > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bourdin at lsu.edu Tue Aug 14 18:24:36 2012 From: bourdin at lsu.edu (Blaise Bourdin) Date: Tue, 14 Aug 2012 18:24:36 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> Message-ID: <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> Hi, On Aug 14, 2012, at 5:58 PM, Barry Smith wrote: > > Blaise, > > You can run with -snes_linesearch_monitor -info -ksp_monitor_true_residual -ksp_converged_reason > > > to get much more information about what is happening and why the line search is failing. Focussing on the relevant part of the output, I get the following for the first SNES step *** Using l2 Residual norms for temp_ solve. 0 KSP preconditioned resid norm 2.352873068990e+00 true resid norm 3.742138023215e+00 ||r(i)||/||b|| 1.000000000000e+00 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS 1 KSP preconditioned resid norm 7.175926524783e-01 true resid norm 8.026926174904e-01 ||r(i)||/||b|| 2.145010719836e-01 2 KSP preconditioned resid norm 4.099791012407e-01 true resid norm 6.219898727406e-01 ||r(i)||/||b|| 1.662124349455e-01 3 KSP preconditioned resid norm 2.769612150659e-01 true resid norm 4.622335508644e-01 ||r(i)||/||b|| 1.235212458752e-01 4 KSP preconditioned resid norm 8.991164116822e-02 true resid norm 1.938840972976e-01 ||r(i)||/||b|| 5.181104921701e-02 5 KSP preconditioned resid norm 1.369794733551e-02 true resid norm 2.867541652138e-02 ||r(i)||/||b|| 7.662843097578e-03 6 KSP preconditioned resid norm 3.522245138600e-03 true resid norm 5.452585588775e-03 ||r(i)||/||b|| 1.457077626466e-03 7 KSP preconditioned resid norm 1.117008651636e-03 true resid norm 1.551905826961e-03 ||r(i)||/||b|| 4.147110067382e-04 8 KSP preconditioned resid norm 5.008635361807e-04 true resid norm 6.226120116381e-04 ||r(i)||/||b|| 1.663786872038e-04 9 KSP preconditioned resid norm 2.079118910844e-04 true resid norm 3.430664466007e-04 ||r(i)||/||b|| 9.167658821571e-05 10 KSP preconditioned resid norm 4.528126074206e-05 true resid norm 9.520866575330e-05 ||r(i)||/||b|| 2.544231804457e-05 11 KSP preconditioned resid norm 8.441137224072e-06 true resid norm 1.519916221879e-05 ||r(i)||/||b|| 4.061625232553e-06 12 KSP preconditioned resid norm 1.839317974157e-06 true resid norm 3.245208227421e-06 ||r(i)||/||b|| 8.672069836252e-07 13 KSP preconditioned resid norm 4.346353491153e-07 true resid norm 7.198101954057e-07 ||r(i)||/||b|| 1.923526580100e-07 14 KSP preconditioned resid norm 6.321413805477e-08 true resid norm 1.280486229700e-07 ||r(i)||/||b|| 3.421803850515e-08 15 KSP preconditioned resid norm 9.029476674935e-09 true resid norm 2.193598397200e-08 ||r(i)||/||b|| 5.861885327562e-09 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 9.029476674935e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 2.352873068990e+00 at iteration 15 Linear solve converged due to CONVERGED_RTOL iterations 15 [0] SNESSolve_LS(): iter=0, linear solve iterations=15 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 4.179425981164e+00 near zero implies inconsistent rhs Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] Line search terminated: lambda = 1, fnorms = 2.19338e-08 [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=2.1933796240714327e-08, ynorm=5.1838420564550498e+00, lssucceed=1 1 SNES Function norm 2.193379624071e-08 [0] SNESDefaultConverged(): Converged due to function norm 2.193379624071e-08 < 3.742138023215e-08 (relative tolerance) SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 *** Using bt Residual norms for temp_ solve. 0 KSP preconditioned resid norm 1.176436534193e+00 true resid norm 3.742138022596e+00 ||r(i)||/||b|| 9.999999998344e-01 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS 1 KSP preconditioned resid norm 3.587963259712e-01 true resid norm 8.026926179905e-01 ||r(i)||/||b|| 2.145010721173e-01 2 KSP preconditioned resid norm 2.049895509618e-01 true resid norm 6.219898720314e-01 ||r(i)||/||b|| 1.662124347560e-01 3 KSP preconditioned resid norm 1.384806072424e-01 true resid norm 4.622335514699e-01 ||r(i)||/||b|| 1.235212460370e-01 4 KSP preconditioned resid norm 4.495582078268e-02 true resid norm 1.938840967382e-01 ||r(i)||/||b|| 5.181104906751e-02 5 KSP preconditioned resid norm 6.848973644691e-03 true resid norm 2.867541656135e-02 ||r(i)||/||b|| 7.662843108259e-03 6 KSP preconditioned resid norm 1.761122593261e-03 true resid norm 5.452585577786e-03 ||r(i)||/||b|| 1.457077623530e-03 7 KSP preconditioned resid norm 5.585043310756e-04 true resid norm 1.551905808041e-03 ||r(i)||/||b|| 4.147110016822e-04 8 KSP preconditioned resid norm 2.504317746421e-04 true resid norm 6.226120013063e-04 ||r(i)||/||b|| 1.663786844429e-04 9 KSP preconditioned resid norm 1.039559493091e-04 true resid norm 3.430664433173e-04 ||r(i)||/||b|| 9.167658733830e-05 10 KSP preconditioned resid norm 2.264063167431e-05 true resid norm 9.520867156897e-05 ||r(i)||/||b|| 2.544231959867e-05 11 KSP preconditioned resid norm 4.220568874641e-06 true resid norm 1.519916304124e-05 ||r(i)||/||b|| 4.061625452335e-06 12 KSP preconditioned resid norm 9.196589910150e-07 true resid norm 3.245208482213e-06 ||r(i)||/||b|| 8.672070517123e-07 13 KSP preconditioned resid norm 2.173176852660e-07 true resid norm 7.198102176806e-07 ||r(i)||/||b|| 1.923526639624e-07 14 KSP preconditioned resid norm 3.160707194729e-08 true resid norm 1.280486368595e-07 ||r(i)||/||b|| 3.421804221680e-08 15 KSP preconditioned resid norm 4.514738683754e-09 true resid norm 2.193598711826e-08 ||r(i)||/||b|| 5.861886168328e-09 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.514738683754e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 1.176436534495e+00 at iteration 15 Linear solve converged due to CONVERGED_RTOL iterations 15 [0] SNESSolve_LS(): iter=0, linear solve iterations=15 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs [0] VecScatterCreate(): Special case: sequential vector general scatter [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 1 SNES Function norm 1.871069011453e+00 As expected, the KSP residuals are exactly the same. I am not sure what to make of [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs. Which RHS is this referring to? the RHS for SNESSolve is 0 (second argument of SNESSolve is PETSC_NULL_OBJECT). Could this mean that there is an incompatibility between my Jacobian and my Function? In all case that diverge, it looks like the gnorm in the linesearch does not decrease. > > If that doesn't help you can send the code and we can play with it. The code is a bit of a pain to build but both Matt and Jed have accounts on my systems. I can arrange to give access to others if necessary. Blaise > > Barry > > > > On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: > >> HI, >> >> I am trying to understand if the following behavior is normal / expected: >> >> I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. >> >> In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. >> >> The relevant code is as follow: >> Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) >> Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) >> Call VecCopy(solTemp,tmpvec,ierr) >> >> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >> >> Call VecCopy(tmpvec,soltemp,ierr) >> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >> >> >> Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. >> >> Blaise >> >> >> >> Below is the sample output using cp as the linesearch type in which case I get the expected convergence: >> Solving time step 1: t= 1.00000E+00 >> 0 SNES Function norm 3.742138023215e+00 >> Line search terminated: lambda = 1, fnorms = 2.1936e-08 >> 1 SNES Function norm 2.193598339906e-08 >> SNES Object:(temp_) 1 MPI processes >> type: ls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=15 >> total number of function evaluations=3 >> KSP Object: (temp_) 1 MPI processes >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (temp_) 1 MPI processes >> type: icc >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> using Manteuffel shift >> matrix ordering: natural >> factor fill ratio given 1, needed 1 >> Factored matrix follows: >> Matrix Object: 1 MPI processes >> type: seqsbaij >> rows=104, cols=104 >> package used to perform factorization: petsc >> total: nonzeros=381, allocated nonzeros=381 >> total number of mallocs used during MatSetValues calls =0 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=104, cols=104 >> total: nonzeros=658, allocated nonzeros=658 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> SNESLineSearch Object: (temp_) 1 MPI processes >> type: cp >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >> maximum iterations=1 >> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >> 0 SNES Function norm 3.742138023215e+00 >> Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] >> Line search terminated: lambda = 2, fnorms = 2.1936e-08 >> 1 SNES Function norm 2.193598717772e-08 >> SNES Object:(temp_) 1 MPI processes >> type: ls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=15 >> total number of function evaluations=3 >> KSP Object: (temp_) 1 MPI processes >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (temp_) 1 MPI processes >> type: icc >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> using Manteuffel shift >> matrix ordering: natural >> factor fill ratio given 1, needed 1 >> Factored matrix follows: >> Matrix Object: 1 MPI processes >> type: seqsbaij >> rows=104, cols=104 >> package used to perform factorization: petsc >> total: nonzeros=381, allocated nonzeros=381 >> total number of mallocs used during MatSetValues calls =0 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=104, cols=104 >> total: nonzeros=658, allocated nonzeros=658 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> SNESLineSearch Object: (temp_) 1 MPI processes >> type: cp >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >> maximum iterations=1 >> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >> >> >> when using the default linesearch (bt), the second SNESSolve fails: >> >> Solving time step 1: t= 1.00000E+00 >> 0 SNES Function norm 3.742138023215e+00 >> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 >> 1 SNES Function norm 2.193598339906e-08 >> SNES Object:(temp_) 1 MPI processes >> type: ls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=15 >> total number of function evaluations=2 >> KSP Object: (temp_) 1 MPI processes >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (temp_) 1 MPI processes >> type: icc >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> using Manteuffel shift >> matrix ordering: natural >> factor fill ratio given 1, needed 1 >> Factored matrix follows: >> Matrix Object: 1 MPI processes >> type: seqsbaij >> rows=104, cols=104 >> package used to perform factorization: petsc >> total: nonzeros=381, allocated nonzeros=381 >> total number of mallocs used during MatSetValues calls =0 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=104, cols=104 >> total: nonzeros=658, allocated nonzeros=658 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> SNESLineSearch Object: (temp_) 1 MPI processes >> type: bt >> interpolation: cubic >> alpha=1.000000e-04 >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >> maximum iterations=40 >> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >> 0 SNES Function norm 3.742138023215e+00 >> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >> 1 SNES Function norm 1.871069011453e+00 >> Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 >> 2 SNES Function norm 1.247379340865e+00 >> Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 >> 3 SNES Function norm 9.355345056231e-01 >> Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 >> 4 SNES Function norm 7.484276044882e-01 >> Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 >> 5 SNES Function norm 6.236896704016e-01 >> Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 >> 6 SNES Function norm 5.345911460556e-01 >> Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 >> 7 SNES Function norm 4.677672528597e-01 >> Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 >> 8 SNES Function norm 4.157931136937e-01 >> Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 >> 9 SNES Function norm 3.742138023528e-01 >> Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 >> 10 SNES Function norm 3.401943657960e-01 >> Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 >> 11 SNES Function norm 3.118448353285e-01 >> Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 >> 12 SNES Function norm 2.878567710844e-01 >> Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 >> 13 SNES Function norm 2.672955731592e-01 >> Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 >> 14 SNES Function norm 2.494758682895e-01 >> Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 >> 15 SNES Function norm 2.338836265275e-01 >> Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 >> 16 SNES Function norm 2.201257661485e-01 >> Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 >> 17 SNES Function norm 2.078965569222e-01 >> Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 >> 18 SNES Function norm 1.969546328772e-01 >> Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 >> 19 SNES Function norm 1.871069012364e-01 >> Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 >> 20 SNES Function norm 1.781970487991e-01 >> Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 >> 21 SNES Function norm 1.700971829468e-01 >> Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 >> 22 SNES Function norm 1.627016532554e-01 >> Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 >> 23 SNES Function norm 1.559224177048e-01 >> Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 >> 24 SNES Function norm 1.496855209981e-01 >> Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 >> 25 SNES Function norm 1.439283855764e-01 >> Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 >> 26 SNES Function norm 1.385977046303e-01 >> Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 >> 27 SNES Function norm 1.336477866088e-01 >> Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 >> 28 SNES Function norm 1.290392422439e-01 >> Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 >> 29 SNES Function norm 1.247379341700e-01 >> Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 >> 30 SNES Function norm 1.207141298427e-01 >> Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 >> 31 SNES Function norm 1.169418132858e-01 >> Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 >> 32 SNES Function norm 1.133981219747e-01 >> Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 >> 33 SNES Function norm 1.100628830937e-01 >> Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 >> 34 SNES Function norm 1.069182292915e-01 >> Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 >> 35 SNES Function norm 1.039482784783e-01 >> Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 >> 36 SNES Function norm 1.011388655469e-01 >> Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 >> 37 SNES Function norm 9.847731645400e-02 >> Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 >> 38 SNES Function norm 9.595225705796e-02 >> Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 >> 39 SNES Function norm 9.355345063171e-02 >> Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 >> 40 SNES Function norm 9.127165915308e-02 >> Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 >> 41 SNES Function norm 8.909852441151e-02 >> Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 >> 42 SNES Function norm 8.702646570443e-02 >> Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 >> 43 SNES Function norm 8.504859148402e-02 >> Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 >> 44 SNES Function norm 8.315862278451e-02 >> Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 >> 45 SNES Function norm 8.135082663716e-02 >> Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 >> 46 SNES Function norm 7.961995798542e-02 >> Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 >> 47 SNES Function norm 7.796120886084e-02 >> Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 >> 48 SNES Function norm 7.637016378216e-02 >> Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 >> 49 SNES Function norm 7.484276050661e-02 >> Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 >> 50 SNES Function norm 7.337525539874e-02 >> SNES Object:(temp_) 1 MPI processes >> type: ls >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=693 >> total number of function evaluations=51 >> KSP Object: (temp_) 1 MPI processes >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (temp_) 1 MPI processes >> type: icc >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> using Manteuffel shift >> matrix ordering: natural >> factor fill ratio given 1, needed 1 >> Factored matrix follows: >> Matrix Object: 1 MPI processes >> type: seqsbaij >> rows=104, cols=104 >> package used to perform factorization: petsc >> total: nonzeros=381, allocated nonzeros=381 >> total number of mallocs used during MatSetValues calls =0 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=104, cols=104 >> total: nonzeros=658, allocated nonzeros=658 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> SNESLineSearch Object: (temp_) 1 MPI processes >> type: bt >> interpolation: cubic >> alpha=1.000000e-04 >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >> maximum iterations=40 >> SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 >> >> >> >> >> >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> > -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Aug 14 18:56:35 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 14 Aug 2012 23:56:35 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> Message-ID: <7237384C7A8F2642A8854B751619FA316F2738C3@exchange.scsolutions.com> Barry, I will install valgrind as one of my own packages on this machine and get back to you later. Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Tuesday, August 14, 2012 3:55 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() Can you run with valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: > Thanks, Matt. > > 1. Yes, I have checked the returned values from x obtained from > MatSolve(F,b,x) > > The norm error check for x is complete for N=75, 2028. > > 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. > > > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range [391]PETSC ERROR: Try > option -start_in_debugger or -on_error_attach_debugger [391]PETSC > ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors [391]PETSC ERROR: likely location of > problem given in stack below [391]PETSC ERROR: --------------------- > Stack Frames ------------------------------------ > [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [391]PETSC ERROR: INSTEAD the line number of the start of the function > [391]PETSC ERROR: is given. > [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/ai > j/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] > MatLUFactorNumeric line 2778 > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interfac > e/matrix.c [391]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [391]PETSC ERROR: Signal received! > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 > 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for > recent updates. > [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [391]PETSC ERROR: See docs/index.html for manual pages. > [391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL > on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 > 2012 [391]PETSC ERROR: Libraries linked from > /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx- > debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 > [391]PETSC ERROR: Configure options > --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib > /librefblas.a > --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/l > ib/libreflapack.a --download-blacs --download-scalapack > --with-mpi-dir=/usr/local/mvapich2/1.7-gnu > --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex > --with-precision=double --with-clanguage=cxx > --with-fortran-kernels=generic --download-mumps > --download-superlu_dist --download-parmetis --download-metis > --with-fortran-interfaces[391]PETSC ERROR: > ---------------------------------------------------------------------- > -- [391]PETSC ERROR: User provided function() line 0 in unknown > directory unknown file > [cli_391]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 > > > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Tuesday, August 14, 2012 3:34 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: > Dear PETSc folks, > > I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: > > 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is > > MatGetOrdering(A,...) > > MatGetFactor(A,...) > > MatLUFactorSymbolic(F, A,...) > > MatLUFactorNumeric(F, A,...) > > MatSolve(F,b,x) > > 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. > > 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: > > > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Signal received! > > ALWAYS send the entire error message. How can we tell anything from a small snippet? > > Since you have [1], this was run in parallel, so you need 3rd party > packages. But you do not seem to be checking return values. Check them > to make sure those packages are installed correctly. > > Matt > > Does anybody have similar experience on that? > > Thanks a lot! > > Jinquan > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Tue Aug 14 21:51:34 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Aug 2012 21:51:34 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> Message-ID: <4BC7C5E3-3355-41F9-B162-7B8F547EAE12@mcs.anl.gov> Likely a bug in our bt. It says it is taking a full step but then seems to take a half-step (based on the result from l2 Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] > [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 > 1 SNES Function norm 1.871069011453e+00 ^^^^^ result as if took a 1/2 step Barry On Aug 14, 2012, at 6:24 PM, Blaise Bourdin wrote: > Hi, > > > On Aug 14, 2012, at 5:58 PM, Barry Smith wrote: > >> >> Blaise, >> >> You can run with -snes_linesearch_monitor -info -ksp_monitor_true_residual -ksp_converged_reason >> >> >> to get much more information about what is happening and why the line search is failing. > Focussing on the relevant part of the output, I get the following for the first SNES step > > *** Using l2 > Residual norms for temp_ solve. > 0 KSP preconditioned resid norm 2.352873068990e+00 true resid norm 3.742138023215e+00 ||r(i)||/||b|| 1.000000000000e+00 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS > 1 KSP preconditioned resid norm 7.175926524783e-01 true resid norm 8.026926174904e-01 ||r(i)||/||b|| 2.145010719836e-01 > 2 KSP preconditioned resid norm 4.099791012407e-01 true resid norm 6.219898727406e-01 ||r(i)||/||b|| 1.662124349455e-01 > 3 KSP preconditioned resid norm 2.769612150659e-01 true resid norm 4.622335508644e-01 ||r(i)||/||b|| 1.235212458752e-01 > 4 KSP preconditioned resid norm 8.991164116822e-02 true resid norm 1.938840972976e-01 ||r(i)||/||b|| 5.181104921701e-02 > 5 KSP preconditioned resid norm 1.369794733551e-02 true resid norm 2.867541652138e-02 ||r(i)||/||b|| 7.662843097578e-03 > 6 KSP preconditioned resid norm 3.522245138600e-03 true resid norm 5.452585588775e-03 ||r(i)||/||b|| 1.457077626466e-03 > 7 KSP preconditioned resid norm 1.117008651636e-03 true resid norm 1.551905826961e-03 ||r(i)||/||b|| 4.147110067382e-04 > 8 KSP preconditioned resid norm 5.008635361807e-04 true resid norm 6.226120116381e-04 ||r(i)||/||b|| 1.663786872038e-04 > 9 KSP preconditioned resid norm 2.079118910844e-04 true resid norm 3.430664466007e-04 ||r(i)||/||b|| 9.167658821571e-05 > 10 KSP preconditioned resid norm 4.528126074206e-05 true resid norm 9.520866575330e-05 ||r(i)||/||b|| 2.544231804457e-05 > 11 KSP preconditioned resid norm 8.441137224072e-06 true resid norm 1.519916221879e-05 ||r(i)||/||b|| 4.061625232553e-06 > 12 KSP preconditioned resid norm 1.839317974157e-06 true resid norm 3.245208227421e-06 ||r(i)||/||b|| 8.672069836252e-07 > 13 KSP preconditioned resid norm 4.346353491153e-07 true resid norm 7.198101954057e-07 ||r(i)||/||b|| 1.923526580100e-07 > 14 KSP preconditioned resid norm 6.321413805477e-08 true resid norm 1.280486229700e-07 ||r(i)||/||b|| 3.421803850515e-08 > 15 KSP preconditioned resid norm 9.029476674935e-09 true resid norm 2.193598397200e-08 ||r(i)||/||b|| 5.861885327562e-09 > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 9.029476674935e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 2.352873068990e+00 at iteration 15 > Linear solve converged due to CONVERGED_RTOL iterations 15 > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 4.179425981164e+00 near zero implies inconsistent rhs > Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] > Line search terminated: lambda = 1, fnorms = 2.19338e-08 > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=2.1933796240714327e-08, ynorm=5.1838420564550498e+00, lssucceed=1 > 1 SNES Function norm 2.193379624071e-08 > [0] SNESDefaultConverged(): Converged due to function norm 2.193379624071e-08 < 3.742138023215e-08 (relative tolerance) > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > > *** Using bt > Residual norms for temp_ solve. > 0 KSP preconditioned resid norm 1.176436534193e+00 true resid norm 3.742138022596e+00 ||r(i)||/||b|| 9.999999998344e-01 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS > 1 KSP preconditioned resid norm 3.587963259712e-01 true resid norm 8.026926179905e-01 ||r(i)||/||b|| 2.145010721173e-01 > 2 KSP preconditioned resid norm 2.049895509618e-01 true resid norm 6.219898720314e-01 ||r(i)||/||b|| 1.662124347560e-01 > 3 KSP preconditioned resid norm 1.384806072424e-01 true resid norm 4.622335514699e-01 ||r(i)||/||b|| 1.235212460370e-01 > 4 KSP preconditioned resid norm 4.495582078268e-02 true resid norm 1.938840967382e-01 ||r(i)||/||b|| 5.181104906751e-02 > 5 KSP preconditioned resid norm 6.848973644691e-03 true resid norm 2.867541656135e-02 ||r(i)||/||b|| 7.662843108259e-03 > 6 KSP preconditioned resid norm 1.761122593261e-03 true resid norm 5.452585577786e-03 ||r(i)||/||b|| 1.457077623530e-03 > 7 KSP preconditioned resid norm 5.585043310756e-04 true resid norm 1.551905808041e-03 ||r(i)||/||b|| 4.147110016822e-04 > 8 KSP preconditioned resid norm 2.504317746421e-04 true resid norm 6.226120013063e-04 ||r(i)||/||b|| 1.663786844429e-04 > 9 KSP preconditioned resid norm 1.039559493091e-04 true resid norm 3.430664433173e-04 ||r(i)||/||b|| 9.167658733830e-05 > 10 KSP preconditioned resid norm 2.264063167431e-05 true resid norm 9.520867156897e-05 ||r(i)||/||b|| 2.544231959867e-05 > 11 KSP preconditioned resid norm 4.220568874641e-06 true resid norm 1.519916304124e-05 ||r(i)||/||b|| 4.061625452335e-06 > 12 KSP preconditioned resid norm 9.196589910150e-07 true resid norm 3.245208482213e-06 ||r(i)||/||b|| 8.672070517123e-07 > 13 KSP preconditioned resid norm 2.173176852660e-07 true resid norm 7.198102176806e-07 ||r(i)||/||b|| 1.923526639624e-07 > 14 KSP preconditioned resid norm 3.160707194729e-08 true resid norm 1.280486368595e-07 ||r(i)||/||b|| 3.421804221680e-08 > 15 KSP preconditioned resid norm 4.514738683754e-09 true resid norm 2.193598711826e-08 ||r(i)||/||b|| 5.861886168328e-09 > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.514738683754e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 1.176436534495e+00 at iteration 15 > Linear solve converged due to CONVERGED_RTOL iterations 15 > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs > [0] VecScatterCreate(): Special case: sequential vector general scatter > [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 > 1 SNES Function norm 1.871069011453e+00 > > > As expected, the KSP residuals are exactly the same. I am not sure what to make of > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs. > Which RHS is this referring to? the RHS for SNESSolve is 0 (second argument of SNESSolve is PETSC_NULL_OBJECT). Could this mean that there is an incompatibility between my Jacobian and my Function? > > In all case that diverge, it looks like the gnorm in the linesearch does not decrease. > >> >> If that doesn't help you can send the code and we can play with it. > > The code is a bit of a pain to build but both Matt and Jed have accounts on my systems. I can arrange to give access to others if necessary. > > Blaise > > >> >> Barry >> >> >> >> On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: >> >>> HI, >>> >>> I am trying to understand if the following behavior is normal / expected: >>> >>> I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. >>> >>> In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. >>> >>> The relevant code is as follow: >>> Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) >>> Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) >>> Call VecCopy(solTemp,tmpvec,ierr) >>> >>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>> >>> Call VecCopy(tmpvec,soltemp,ierr) >>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>> >>> >>> Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. >>> >>> Blaise >>> >>> >>> >>> Below is the sample output using cp as the linesearch type in which case I get the expected convergence: >>> Solving time step 1: t= 1.00000E+00 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search terminated: lambda = 1, fnorms = 2.1936e-08 >>> 1 SNES Function norm 2.193598339906e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=3 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: cp >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=1 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] >>> Line search terminated: lambda = 2, fnorms = 2.1936e-08 >>> 1 SNES Function norm 2.193598717772e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=3 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: cp >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=1 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> >>> >>> when using the default linesearch (bt), the second SNESSolve fails: >>> >>> Solving time step 1: t= 1.00000E+00 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 >>> 1 SNES Function norm 2.193598339906e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=2 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: bt >>> interpolation: cubic >>> alpha=1.000000e-04 >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=40 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >>> 1 SNES Function norm 1.871069011453e+00 >>> Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 >>> 2 SNES Function norm 1.247379340865e+00 >>> Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 >>> 3 SNES Function norm 9.355345056231e-01 >>> Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 >>> 4 SNES Function norm 7.484276044882e-01 >>> Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 >>> 5 SNES Function norm 6.236896704016e-01 >>> Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 >>> 6 SNES Function norm 5.345911460556e-01 >>> Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 >>> 7 SNES Function norm 4.677672528597e-01 >>> Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 >>> 8 SNES Function norm 4.157931136937e-01 >>> Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 >>> 9 SNES Function norm 3.742138023528e-01 >>> Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 >>> 10 SNES Function norm 3.401943657960e-01 >>> Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 >>> 11 SNES Function norm 3.118448353285e-01 >>> Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 >>> 12 SNES Function norm 2.878567710844e-01 >>> Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 >>> 13 SNES Function norm 2.672955731592e-01 >>> Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 >>> 14 SNES Function norm 2.494758682895e-01 >>> Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 >>> 15 SNES Function norm 2.338836265275e-01 >>> Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 >>> 16 SNES Function norm 2.201257661485e-01 >>> Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 >>> 17 SNES Function norm 2.078965569222e-01 >>> Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 >>> 18 SNES Function norm 1.969546328772e-01 >>> Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 >>> 19 SNES Function norm 1.871069012364e-01 >>> Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 >>> 20 SNES Function norm 1.781970487991e-01 >>> Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 >>> 21 SNES Function norm 1.700971829468e-01 >>> Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 >>> 22 SNES Function norm 1.627016532554e-01 >>> Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 >>> 23 SNES Function norm 1.559224177048e-01 >>> Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 >>> 24 SNES Function norm 1.496855209981e-01 >>> Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 >>> 25 SNES Function norm 1.439283855764e-01 >>> Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 >>> 26 SNES Function norm 1.385977046303e-01 >>> Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 >>> 27 SNES Function norm 1.336477866088e-01 >>> Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 >>> 28 SNES Function norm 1.290392422439e-01 >>> Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 >>> 29 SNES Function norm 1.247379341700e-01 >>> Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 >>> 30 SNES Function norm 1.207141298427e-01 >>> Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 >>> 31 SNES Function norm 1.169418132858e-01 >>> Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 >>> 32 SNES Function norm 1.133981219747e-01 >>> Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 >>> 33 SNES Function norm 1.100628830937e-01 >>> Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 >>> 34 SNES Function norm 1.069182292915e-01 >>> Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 >>> 35 SNES Function norm 1.039482784783e-01 >>> Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 >>> 36 SNES Function norm 1.011388655469e-01 >>> Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 >>> 37 SNES Function norm 9.847731645400e-02 >>> Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 >>> 38 SNES Function norm 9.595225705796e-02 >>> Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 >>> 39 SNES Function norm 9.355345063171e-02 >>> Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 >>> 40 SNES Function norm 9.127165915308e-02 >>> Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 >>> 41 SNES Function norm 8.909852441151e-02 >>> Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 >>> 42 SNES Function norm 8.702646570443e-02 >>> Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 >>> 43 SNES Function norm 8.504859148402e-02 >>> Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 >>> 44 SNES Function norm 8.315862278451e-02 >>> Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 >>> 45 SNES Function norm 8.135082663716e-02 >>> Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 >>> 46 SNES Function norm 7.961995798542e-02 >>> Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 >>> 47 SNES Function norm 7.796120886084e-02 >>> Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 >>> 48 SNES Function norm 7.637016378216e-02 >>> Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 >>> 49 SNES Function norm 7.484276050661e-02 >>> Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 >>> 50 SNES Function norm 7.337525539874e-02 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=693 >>> total number of function evaluations=51 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: bt >>> interpolation: cubic >>> alpha=1.000000e-04 >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=40 >>> SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 >>> >>> >>> >>> >>> >>> -- >>> Department of Mathematics and Center for Computation & Technology >>> Louisiana State University, Baton Rouge, LA 70803, USA >>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >>> >>> >>> >>> >>> >>> >>> >> > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > > From bsmith at mcs.anl.gov Tue Aug 14 21:54:23 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Aug 2012 21:54:23 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F2738A5@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> <7237384C7A8F2642A8854B751619FA316F2738A5@exchange.scsolutions.com> Message-ID: <500F4234-3C1C-4F81-9AF5-3F30EC4D0A24@mcs.anl.gov> On Aug 14, 2012, at 6:05 PM, Jinquan Zhong wrote: > Barry, > > The machine I ran this program does not have valgrind. > > Another interesting observation is that when I ran the same three matrices using PETSc3.2. MatLUFactorNumeric() hanged up even on N=75, 2028 till I specified -mat_superlu_dist_colperm. However, MatLUFactorNumeric() didn't work for N=21180 either even I used > > -mat_superlu_dist_rowperm NATURAL -mat_superlu_dist_colperm NATURAL -mat_superlu_dist_parsymbfact YES > > I suspect that there is something incompatible in the factored matrix from superLU-dist to be used MatLUFactorNumeric() in PETSc3.2. Although PETSc 3.3 fixed this issue for matrix with small N, however, this issue relapsed for large N in PETSc3.3. It is using Superlu_dist for this factorization (and that version changed with PETSc 3.3) the problem is with Superlu_Dist not PETSc. valgrind will likely find an error in SuperLU_dist Barry > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Tuesday, August 14, 2012 3:55 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > > Can you run with valgrind > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: > >> Thanks, Matt. >> >> 1. Yes, I have checked the returned values from x obtained from >> MatSolve(F,b,x) >> >> The norm error check for x is complete for N=75, 2028. >> >> 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. >> >> >> [391]PETSC ERROR: >> ---------------------------------------------------------------------- >> -- [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >> Violation, probably memory access out of range [391]PETSC ERROR: Try >> option -start_in_debugger or -on_error_attach_debugger [391]PETSC >> ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors [391]PETSC ERROR: likely location of >> problem given in stack below [391]PETSC ERROR: --------------------- >> Stack Frames ------------------------------------ >> [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [391]PETSC ERROR: INSTEAD the line number of the start of the function >> [391]PETSC ERROR: is given. >> [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/ai >> j/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] >> MatLUFactorNumeric line 2778 >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interfac >> e/matrix.c [391]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [391]PETSC ERROR: Signal received! >> [391]PETSC ERROR: >> ---------------------------------------------------------------------- >> -- [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >> 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for >> recent updates. >> [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [391]PETSC ERROR: See docs/index.html for manual pages. >> [391]PETSC ERROR: >> ---------------------------------------------------------------------- >> -- [391]PETSC ERROR: /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL >> on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 >> 2012 [391]PETSC ERROR: Libraries linked from >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx- >> debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 2012 >> [391]PETSC ERROR: Configure options >> --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/lib >> /librefblas.a >> --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/l >> ib/libreflapack.a --download-blacs --download-scalapack >> --with-mpi-dir=/usr/local/mvapich2/1.7-gnu >> --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex >> --with-precision=double --with-clanguage=cxx >> --with-fortran-kernels=generic --download-mumps >> --download-superlu_dist --download-parmetis --download-metis >> --with-fortran-interfaces[391]PETSC ERROR: >> ---------------------------------------------------------------------- >> -- [391]PETSC ERROR: User provided function() line 0 in unknown >> directory unknown file >> [cli_391]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 >> >> >> From: petsc-users-bounces at mcs.anl.gov >> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Tuesday, August 14, 2012 3:34 PM >> To: PETSc users list >> Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() >> >> On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: >> Dear PETSc folks, >> >> I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: >> >> 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is >> >> MatGetOrdering(A,...) >> >> MatGetFactor(A,...) >> >> MatLUFactorSymbolic(F, A,...) >> >> MatLUFactorNumeric(F, A,...) >> >> MatSolve(F,b,x) >> >> 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. >> >> 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: >> >> >> >> [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Signal received! >> >> ALWAYS send the entire error message. How can we tell anything from a small snippet? >> >> Since you have [1], this was run in parallel, so you need 3rd party >> packages. But you do not seem to be checking return values. Check them >> to make sure those packages are installed correctly. >> >> Matt >> >> Does anybody have similar experience on that? >> >> Thanks a lot! >> >> Jinquan >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From prbrune at gmail.com Tue Aug 14 21:58:04 2012 From: prbrune at gmail.com (Peter Brune) Date: Tue, 14 Aug 2012 21:58:04 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: <4BC7C5E3-3355-41F9-B162-7B8F547EAE12@mcs.anl.gov> References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> <4BC7C5E3-3355-41F9-B162-7B8F547EAE12@mcs.anl.gov> Message-ID: Most likely not a bug; more like a feature. BT is meant to take the full step when the descent condition is fine. The descent condition can be fine without making a ton of progress. Taking the full step avoids a lot of work (extra function evaluations) when the solve is working right, but messes up hard when it isn't. I'm wondering why your problem is doing this. The question is, therefore, why is 2x the optimum step? For 1D problems this is explained by classical numerical analysis; the root the Newton's method is finding has multiplicity 2. I'm wondering what exactly would exhibit this behavior in multiple dimensions for a linear problem. - Peter On Tue, Aug 14, 2012 at 9:51 PM, Barry Smith wrote: > > Likely a bug in our bt. It says it is taking a full step but then seems > to take a half-step (based on the result from l2 Line search: lambdas > = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] > > > [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm > 1.871069011453e+00 > > Line search: Using full step: fnorm 3.742138023215e+00 gnorm > 1.871069011453e+00 > > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, > gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 > > 1 SNES Function norm 1.871069011453e+00 > > ^^^^^ result as > if took a 1/2 step > > Barry > > > > On Aug 14, 2012, at 6:24 PM, Blaise Bourdin wrote: > > > Hi, > > > > > > On Aug 14, 2012, at 5:58 PM, Barry Smith wrote: > > > >> > >> Blaise, > >> > >> You can run with -snes_linesearch_monitor -info > -ksp_monitor_true_residual -ksp_converged_reason > >> > >> > >> to get much more information about what is happening and why the line > search is failing. > > Focussing on the relevant part of the output, I get the following for > the first SNES step > > > > *** Using l2 > > Residual norms for temp_ solve. > > 0 KSP preconditioned resid norm 2.352873068990e+00 true resid norm > 3.742138023215e+00 ||r(i)||/||b|| 1.000000000000e+00 > > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > > 1 KSP preconditioned resid norm 7.175926524783e-01 true resid norm > 8.026926174904e-01 ||r(i)||/||b|| 2.145010719836e-01 > > 2 KSP preconditioned resid norm 4.099791012407e-01 true resid norm > 6.219898727406e-01 ||r(i)||/||b|| 1.662124349455e-01 > > 3 KSP preconditioned resid norm 2.769612150659e-01 true resid norm > 4.622335508644e-01 ||r(i)||/||b|| 1.235212458752e-01 > > 4 KSP preconditioned resid norm 8.991164116822e-02 true resid norm > 1.938840972976e-01 ||r(i)||/||b|| 5.181104921701e-02 > > 5 KSP preconditioned resid norm 1.369794733551e-02 true resid norm > 2.867541652138e-02 ||r(i)||/||b|| 7.662843097578e-03 > > 6 KSP preconditioned resid norm 3.522245138600e-03 true resid norm > 5.452585588775e-03 ||r(i)||/||b|| 1.457077626466e-03 > > 7 KSP preconditioned resid norm 1.117008651636e-03 true resid norm > 1.551905826961e-03 ||r(i)||/||b|| 4.147110067382e-04 > > 8 KSP preconditioned resid norm 5.008635361807e-04 true resid norm > 6.226120116381e-04 ||r(i)||/||b|| 1.663786872038e-04 > > 9 KSP preconditioned resid norm 2.079118910844e-04 true resid norm > 3.430664466007e-04 ||r(i)||/||b|| 9.167658821571e-05 > > 10 KSP preconditioned resid norm 4.528126074206e-05 true resid norm > 9.520866575330e-05 ||r(i)||/||b|| 2.544231804457e-05 > > 11 KSP preconditioned resid norm 8.441137224072e-06 true resid norm > 1.519916221879e-05 ||r(i)||/||b|| 4.061625232553e-06 > > 12 KSP preconditioned resid norm 1.839317974157e-06 true resid norm > 3.245208227421e-06 ||r(i)||/||b|| 8.672069836252e-07 > > 13 KSP preconditioned resid norm 4.346353491153e-07 true resid norm > 7.198101954057e-07 ||r(i)||/||b|| 1.923526580100e-07 > > 14 KSP preconditioned resid norm 6.321413805477e-08 true resid norm > 1.280486229700e-07 ||r(i)||/||b|| 3.421803850515e-08 > > 15 KSP preconditioned resid norm 9.029476674935e-09 true resid norm > 2.193598397200e-08 ||r(i)||/||b|| 5.861885327562e-09 > > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm > 9.029476674935e-09 is less than relative tolerance 1.000000000000e-08 times > initial right hand side norm 2.352873068990e+00 at iteration 15 > > Linear solve converged due to CONVERGED_RTOL iterations 15 > > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| > 4.179425981164e+00 near zero implies inconsistent rhs > > Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, > 3.74214] > > Line search terminated: lambda = 1, fnorms = 2.19338e-08 > > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, > gnorm=2.1933796240714327e-08, ynorm=5.1838420564550498e+00, lssucceed=1 > > 1 SNES Function norm 2.193379624071e-08 > > [0] SNESDefaultConverged(): Converged due to function norm > 2.193379624071e-08 < 3.742138023215e-08 (relative tolerance) > > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > > > > *** Using bt > > Residual norms for temp_ solve. > > 0 KSP preconditioned resid norm 1.176436534193e+00 true resid norm > 3.742138022596e+00 ||r(i)||/||b|| 9.999999998344e-01 > > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > > 1 KSP preconditioned resid norm 3.587963259712e-01 true resid norm > 8.026926179905e-01 ||r(i)||/||b|| 2.145010721173e-01 > > 2 KSP preconditioned resid norm 2.049895509618e-01 true resid norm > 6.219898720314e-01 ||r(i)||/||b|| 1.662124347560e-01 > > 3 KSP preconditioned resid norm 1.384806072424e-01 true resid norm > 4.622335514699e-01 ||r(i)||/||b|| 1.235212460370e-01 > > 4 KSP preconditioned resid norm 4.495582078268e-02 true resid norm > 1.938840967382e-01 ||r(i)||/||b|| 5.181104906751e-02 > > 5 KSP preconditioned resid norm 6.848973644691e-03 true resid norm > 2.867541656135e-02 ||r(i)||/||b|| 7.662843108259e-03 > > 6 KSP preconditioned resid norm 1.761122593261e-03 true resid norm > 5.452585577786e-03 ||r(i)||/||b|| 1.457077623530e-03 > > 7 KSP preconditioned resid norm 5.585043310756e-04 true resid norm > 1.551905808041e-03 ||r(i)||/||b|| 4.147110016822e-04 > > 8 KSP preconditioned resid norm 2.504317746421e-04 true resid norm > 6.226120013063e-04 ||r(i)||/||b|| 1.663786844429e-04 > > 9 KSP preconditioned resid norm 1.039559493091e-04 true resid norm > 3.430664433173e-04 ||r(i)||/||b|| 9.167658733830e-05 > > 10 KSP preconditioned resid norm 2.264063167431e-05 true resid norm > 9.520867156897e-05 ||r(i)||/||b|| 2.544231959867e-05 > > 11 KSP preconditioned resid norm 4.220568874641e-06 true resid norm > 1.519916304124e-05 ||r(i)||/||b|| 4.061625452335e-06 > > 12 KSP preconditioned resid norm 9.196589910150e-07 true resid norm > 3.245208482213e-06 ||r(i)||/||b|| 8.672070517123e-07 > > 13 KSP preconditioned resid norm 2.173176852660e-07 true resid norm > 7.198102176806e-07 ||r(i)||/||b|| 1.923526639624e-07 > > 14 KSP preconditioned resid norm 3.160707194729e-08 true resid norm > 1.280486368595e-07 ||r(i)||/||b|| 3.421804221680e-08 > > 15 KSP preconditioned resid norm 4.514738683754e-09 true resid norm > 2.193598711826e-08 ||r(i)||/||b|| 5.861886168328e-09 > > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm > 4.514738683754e-09 is less than relative tolerance 1.000000000000e-08 times > initial right hand side norm 1.176436534495e+00 at iteration 15 > > Linear solve converged due to CONVERGED_RTOL iterations 15 > > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| > 8.358851664967e+00 near zero implies inconsistent rhs > > [0] VecScatterCreate(): Special case: sequential vector general scatter > > [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm > 1.871069011453e+00 > > Line search: Using full step: fnorm 3.742138023215e+00 gnorm > 1.871069011453e+00 > > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, > gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 > > 1 SNES Function norm 1.871069011453e+00 > > > > > > As expected, the KSP residuals are exactly the same. I am not sure what > to make of > > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| > 8.358851664967e+00 near zero implies inconsistent rhs. > > Which RHS is this referring to? the RHS for SNESSolve is 0 (second > argument of SNESSolve is PETSC_NULL_OBJECT). Could this mean that there is > an incompatibility between my Jacobian and my Function? > > > > In all case that diverge, it looks like the gnorm in the linesearch does > not decrease. > > > >> > >> If that doesn't help you can send the code and we can play with it. > > > > The code is a bit of a pain to build but both Matt and Jed have accounts > on my systems. I can arrange to give access to others if necessary. > > > > Blaise > > > > > >> > >> Barry > >> > >> > >> > >> On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: > >> > >>> HI, > >>> > >>> I am trying to understand if the following behavior is normal / > expected: > >>> > >>> I am solving a quasi-static evolution where at each time step, > SNESSolve is called. My validation problem is a _static_ problem with 2 > time steps (i.e. 2 successive calls to SNESSolve with the same operator, > jacobian, and right hand side). Furthermore, the problem is linear, so that > the Jacobian is constant. I even reset the solution vector to the same > value at each time step. > >>> > >>> In this setting, I am expecting that at each time step, each SNESSolve > should converge in exactly one iteration no matter what linesearch / snes > type I use. Indeed, when setting the linesearch to l2 or cp, this is what I > get. However, for all other choices, the second call to SNESSolve fails to > converge with a SNESConvergedReason of -5 or -6. > >>> > >>> The relevant code is as follow: > >>> Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) > >>> Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) > >>> Call VecCopy(solTemp,tmpvec,ierr) > >>> > >>> Call > SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) > >>> > >>> Call VecCopy(tmpvec,soltemp,ierr) > >>> Call > SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) > >>> > >>> > >>> Is this expected? I tried to call SNESLineSearchReset before the > second call to SNESSolve, but this does not seem to have any effect. > >>> > >>> Blaise > >>> > >>> > >>> > >>> Below is the sample output using cp as the linesearch type in which > case I get the expected convergence: > >>> Solving time step 1: t= 1.00000E+00 > >>> 0 SNES Function norm 3.742138023215e+00 > >>> Line search terminated: lambda = 1, fnorms = 2.1936e-08 > >>> 1 SNES Function norm 2.193598339906e-08 > >>> SNES Object:(temp_) 1 MPI processes > >>> type: ls > >>> maximum iterations=50, maximum function evaluations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > >>> total number of linear solver iterations=15 > >>> total number of function evaluations=3 > >>> KSP Object: (temp_) 1 MPI processes > >>> type: cg > >>> maximum iterations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > >>> left preconditioning > >>> using nonzero initial guess > >>> using PRECONDITIONED norm type for convergence test > >>> PC Object: (temp_) 1 MPI processes > >>> type: icc > >>> 0 levels of fill > >>> tolerance for zero pivot 2.22045e-14 > >>> using Manteuffel shift > >>> matrix ordering: natural > >>> factor fill ratio given 1, needed 1 > >>> Factored matrix follows: > >>> Matrix Object: 1 MPI processes > >>> type: seqsbaij > >>> rows=104, cols=104 > >>> package used to perform factorization: petsc > >>> total: nonzeros=381, allocated nonzeros=381 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> linear system matrix = precond matrix: > >>> Matrix Object: 1 MPI processes > >>> type: seqaij > >>> rows=104, cols=104 > >>> total: nonzeros=658, allocated nonzeros=658 > >>> total number of mallocs used during MatSetValues calls =0 > >>> not using I-node routines > >>> SNESLineSearch Object: (temp_) 1 MPI processes > >>> type: cp > >>> maxstep=1.000000e+08, minlambda=1.000000e-12 > >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > >>> maximum iterations=1 > >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > >>> 0 SNES Function norm 3.742138023215e+00 > >>> Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] > >>> Line search terminated: lambda = 2, fnorms = 2.1936e-08 > >>> 1 SNES Function norm 2.193598717772e-08 > >>> SNES Object:(temp_) 1 MPI processes > >>> type: ls > >>> maximum iterations=50, maximum function evaluations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > >>> total number of linear solver iterations=15 > >>> total number of function evaluations=3 > >>> KSP Object: (temp_) 1 MPI processes > >>> type: cg > >>> maximum iterations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > >>> left preconditioning > >>> using nonzero initial guess > >>> using PRECONDITIONED norm type for convergence test > >>> PC Object: (temp_) 1 MPI processes > >>> type: icc > >>> 0 levels of fill > >>> tolerance for zero pivot 2.22045e-14 > >>> using Manteuffel shift > >>> matrix ordering: natural > >>> factor fill ratio given 1, needed 1 > >>> Factored matrix follows: > >>> Matrix Object: 1 MPI processes > >>> type: seqsbaij > >>> rows=104, cols=104 > >>> package used to perform factorization: petsc > >>> total: nonzeros=381, allocated nonzeros=381 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> linear system matrix = precond matrix: > >>> Matrix Object: 1 MPI processes > >>> type: seqaij > >>> rows=104, cols=104 > >>> total: nonzeros=658, allocated nonzeros=658 > >>> total number of mallocs used during MatSetValues calls =0 > >>> not using I-node routines > >>> SNESLineSearch Object: (temp_) 1 MPI processes > >>> type: cp > >>> maxstep=1.000000e+08, minlambda=1.000000e-12 > >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > >>> maximum iterations=1 > >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > >>> > >>> > >>> when using the default linesearch (bt), the second SNESSolve fails: > >>> > >>> Solving time step 1: t= 1.00000E+00 > >>> 0 SNES Function norm 3.742138023215e+00 > >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm > 2.193598339906e-08 > >>> 1 SNES Function norm 2.193598339906e-08 > >>> SNES Object:(temp_) 1 MPI processes > >>> type: ls > >>> maximum iterations=50, maximum function evaluations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > >>> total number of linear solver iterations=15 > >>> total number of function evaluations=2 > >>> KSP Object: (temp_) 1 MPI processes > >>> type: cg > >>> maximum iterations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > >>> left preconditioning > >>> using nonzero initial guess > >>> using PRECONDITIONED norm type for convergence test > >>> PC Object: (temp_) 1 MPI processes > >>> type: icc > >>> 0 levels of fill > >>> tolerance for zero pivot 2.22045e-14 > >>> using Manteuffel shift > >>> matrix ordering: natural > >>> factor fill ratio given 1, needed 1 > >>> Factored matrix follows: > >>> Matrix Object: 1 MPI processes > >>> type: seqsbaij > >>> rows=104, cols=104 > >>> package used to perform factorization: petsc > >>> total: nonzeros=381, allocated nonzeros=381 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> linear system matrix = precond matrix: > >>> Matrix Object: 1 MPI processes > >>> type: seqaij > >>> rows=104, cols=104 > >>> total: nonzeros=658, allocated nonzeros=658 > >>> total number of mallocs used during MatSetValues calls =0 > >>> not using I-node routines > >>> SNESLineSearch Object: (temp_) 1 MPI processes > >>> type: bt > >>> interpolation: cubic > >>> alpha=1.000000e-04 > >>> maxstep=1.000000e+08, minlambda=1.000000e-12 > >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > >>> maximum iterations=40 > >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > >>> 0 SNES Function norm 3.742138023215e+00 > >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm > 1.871069011453e+00 > >>> 1 SNES Function norm 1.871069011453e+00 > >>> Line search: Using full step: fnorm 1.871069011453e+00 gnorm > 1.247379340865e+00 > >>> 2 SNES Function norm 1.247379340865e+00 > >>> Line search: Using full step: fnorm 1.247379340865e+00 gnorm > 9.355345056231e-01 > >>> 3 SNES Function norm 9.355345056231e-01 > >>> Line search: Using full step: fnorm 9.355345056231e-01 gnorm > 7.484276044882e-01 > >>> 4 SNES Function norm 7.484276044882e-01 > >>> Line search: Using full step: fnorm 7.484276044882e-01 gnorm > 6.236896704016e-01 > >>> 5 SNES Function norm 6.236896704016e-01 > >>> Line search: Using full step: fnorm 6.236896704016e-01 gnorm > 5.345911460556e-01 > >>> 6 SNES Function norm 5.345911460556e-01 > >>> Line search: Using full step: fnorm 5.345911460556e-01 gnorm > 4.677672528597e-01 > >>> 7 SNES Function norm 4.677672528597e-01 > >>> Line search: Using full step: fnorm 4.677672528597e-01 gnorm > 4.157931136937e-01 > >>> 8 SNES Function norm 4.157931136937e-01 > >>> Line search: Using full step: fnorm 4.157931136937e-01 gnorm > 3.742138023528e-01 > >>> 9 SNES Function norm 3.742138023528e-01 > >>> Line search: Using full step: fnorm 3.742138023528e-01 gnorm > 3.401943657960e-01 > >>> 10 SNES Function norm 3.401943657960e-01 > >>> Line search: Using full step: fnorm 3.401943657960e-01 gnorm > 3.118448353285e-01 > >>> 11 SNES Function norm 3.118448353285e-01 > >>> Line search: Using full step: fnorm 3.118448353285e-01 gnorm > 2.878567710844e-01 > >>> 12 SNES Function norm 2.878567710844e-01 > >>> Line search: Using full step: fnorm 2.878567710844e-01 gnorm > 2.672955731592e-01 > >>> 13 SNES Function norm 2.672955731592e-01 > >>> Line search: Using full step: fnorm 2.672955731592e-01 gnorm > 2.494758682895e-01 > >>> 14 SNES Function norm 2.494758682895e-01 > >>> Line search: Using full step: fnorm 2.494758682895e-01 gnorm > 2.338836265275e-01 > >>> 15 SNES Function norm 2.338836265275e-01 > >>> Line search: Using full step: fnorm 2.338836265275e-01 gnorm > 2.201257661485e-01 > >>> 16 SNES Function norm 2.201257661485e-01 > >>> Line search: Using full step: fnorm 2.201257661485e-01 gnorm > 2.078965569222e-01 > >>> 17 SNES Function norm 2.078965569222e-01 > >>> Line search: Using full step: fnorm 2.078965569222e-01 gnorm > 1.969546328772e-01 > >>> 18 SNES Function norm 1.969546328772e-01 > >>> Line search: Using full step: fnorm 1.969546328772e-01 gnorm > 1.871069012364e-01 > >>> 19 SNES Function norm 1.871069012364e-01 > >>> Line search: Using full step: fnorm 1.871069012364e-01 gnorm > 1.781970487991e-01 > >>> 20 SNES Function norm 1.781970487991e-01 > >>> Line search: Using full step: fnorm 1.781970487991e-01 gnorm > 1.700971829468e-01 > >>> 21 SNES Function norm 1.700971829468e-01 > >>> Line search: Using full step: fnorm 1.700971829468e-01 gnorm > 1.627016532554e-01 > >>> 22 SNES Function norm 1.627016532554e-01 > >>> Line search: Using full step: fnorm 1.627016532554e-01 gnorm > 1.559224177048e-01 > >>> 23 SNES Function norm 1.559224177048e-01 > >>> Line search: Using full step: fnorm 1.559224177048e-01 gnorm > 1.496855209981e-01 > >>> 24 SNES Function norm 1.496855209981e-01 > >>> Line search: Using full step: fnorm 1.496855209981e-01 gnorm > 1.439283855764e-01 > >>> 25 SNES Function norm 1.439283855764e-01 > >>> Line search: Using full step: fnorm 1.439283855764e-01 gnorm > 1.385977046303e-01 > >>> 26 SNES Function norm 1.385977046303e-01 > >>> Line search: Using full step: fnorm 1.385977046303e-01 gnorm > 1.336477866088e-01 > >>> 27 SNES Function norm 1.336477866088e-01 > >>> Line search: Using full step: fnorm 1.336477866088e-01 gnorm > 1.290392422439e-01 > >>> 28 SNES Function norm 1.290392422439e-01 > >>> Line search: Using full step: fnorm 1.290392422439e-01 gnorm > 1.247379341700e-01 > >>> 29 SNES Function norm 1.247379341700e-01 > >>> Line search: Using full step: fnorm 1.247379341700e-01 gnorm > 1.207141298427e-01 > >>> 30 SNES Function norm 1.207141298427e-01 > >>> Line search: Using full step: fnorm 1.207141298427e-01 gnorm > 1.169418132858e-01 > >>> 31 SNES Function norm 1.169418132858e-01 > >>> Line search: Using full step: fnorm 1.169418132858e-01 gnorm > 1.133981219747e-01 > >>> 32 SNES Function norm 1.133981219747e-01 > >>> Line search: Using full step: fnorm 1.133981219747e-01 gnorm > 1.100628830937e-01 > >>> 33 SNES Function norm 1.100628830937e-01 > >>> Line search: Using full step: fnorm 1.100628830937e-01 gnorm > 1.069182292915e-01 > >>> 34 SNES Function norm 1.069182292915e-01 > >>> Line search: Using full step: fnorm 1.069182292915e-01 gnorm > 1.039482784783e-01 > >>> 35 SNES Function norm 1.039482784783e-01 > >>> Line search: Using full step: fnorm 1.039482784783e-01 gnorm > 1.011388655469e-01 > >>> 36 SNES Function norm 1.011388655469e-01 > >>> Line search: Using full step: fnorm 1.011388655469e-01 gnorm > 9.847731645400e-02 > >>> 37 SNES Function norm 9.847731645400e-02 > >>> Line search: Using full step: fnorm 9.847731645400e-02 gnorm > 9.595225705796e-02 > >>> 38 SNES Function norm 9.595225705796e-02 > >>> Line search: Using full step: fnorm 9.595225705796e-02 gnorm > 9.355345063171e-02 > >>> 39 SNES Function norm 9.355345063171e-02 > >>> Line search: Using full step: fnorm 9.355345063171e-02 gnorm > 9.127165915308e-02 > >>> 40 SNES Function norm 9.127165915308e-02 > >>> Line search: Using full step: fnorm 9.127165915308e-02 gnorm > 8.909852441151e-02 > >>> 41 SNES Function norm 8.909852441151e-02 > >>> Line search: Using full step: fnorm 8.909852441151e-02 gnorm > 8.702646570443e-02 > >>> 42 SNES Function norm 8.702646570443e-02 > >>> Line search: Using full step: fnorm 8.702646570443e-02 gnorm > 8.504859148402e-02 > >>> 43 SNES Function norm 8.504859148402e-02 > >>> Line search: Using full step: fnorm 8.504859148402e-02 gnorm > 8.315862278451e-02 > >>> 44 SNES Function norm 8.315862278451e-02 > >>> Line search: Using full step: fnorm 8.315862278451e-02 gnorm > 8.135082663716e-02 > >>> 45 SNES Function norm 8.135082663716e-02 > >>> Line search: Using full step: fnorm 8.135082663716e-02 gnorm > 7.961995798542e-02 > >>> 46 SNES Function norm 7.961995798542e-02 > >>> Line search: Using full step: fnorm 7.961995798542e-02 gnorm > 7.796120886084e-02 > >>> 47 SNES Function norm 7.796120886084e-02 > >>> Line search: Using full step: fnorm 7.796120886084e-02 gnorm > 7.637016378216e-02 > >>> 48 SNES Function norm 7.637016378216e-02 > >>> Line search: Using full step: fnorm 7.637016378216e-02 gnorm > 7.484276050661e-02 > >>> 49 SNES Function norm 7.484276050661e-02 > >>> Line search: Using full step: fnorm 7.484276050661e-02 gnorm > 7.337525539874e-02 > >>> 50 SNES Function norm 7.337525539874e-02 > >>> SNES Object:(temp_) 1 MPI processes > >>> type: ls > >>> maximum iterations=50, maximum function evaluations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > >>> total number of linear solver iterations=693 > >>> total number of function evaluations=51 > >>> KSP Object: (temp_) 1 MPI processes > >>> type: cg > >>> maximum iterations=10000 > >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > >>> left preconditioning > >>> using nonzero initial guess > >>> using PRECONDITIONED norm type for convergence test > >>> PC Object: (temp_) 1 MPI processes > >>> type: icc > >>> 0 levels of fill > >>> tolerance for zero pivot 2.22045e-14 > >>> using Manteuffel shift > >>> matrix ordering: natural > >>> factor fill ratio given 1, needed 1 > >>> Factored matrix follows: > >>> Matrix Object: 1 MPI processes > >>> type: seqsbaij > >>> rows=104, cols=104 > >>> package used to perform factorization: petsc > >>> total: nonzeros=381, allocated nonzeros=381 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> linear system matrix = precond matrix: > >>> Matrix Object: 1 MPI processes > >>> type: seqaij > >>> rows=104, cols=104 > >>> total: nonzeros=658, allocated nonzeros=658 > >>> total number of mallocs used during MatSetValues calls =0 > >>> not using I-node routines > >>> SNESLineSearch Object: (temp_) 1 MPI processes > >>> type: bt > >>> interpolation: cubic > >>> alpha=1.000000e-04 > >>> maxstep=1.000000e+08, minlambda=1.000000e-12 > >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > >>> maximum iterations=40 > >>> SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Department of Mathematics and Center for Computation & Technology > >>> Louisiana State University, Baton Rouge, LA 70803, USA > >>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >> > > > > -- > > Department of Mathematics and Center for Computation & Technology > > Louisiana State University, Baton Rouge, LA 70803, USA > > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Aug 15 10:06:30 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 15 Aug 2012 10:06:30 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> Message-ID: Jinquan: > ** ** > > I have a strange observation on using MatLUFactorNumeric() for dense > matrices at different order N. Here is the situation I have: > Superlu and mumps are sparse direct solvers, not for dense matrices. > **1. **I use ./src/mat/tests/ex137.c as an example to direct PETSc > in > petsc-3.3/src/mat/examples/tests/ex137.c is a "Tests MatCreateMPISBAIJWithArrays()." Which version of petsc do you use? > selecting superLU-dist and mumps. The calling sequence is **** > > MatGetOrdering(A,?)**** > > MatGetFactor(A,?)**** > > MatLUFactorSymbolic(F, A,?)**** > > *MatLUFactorNumeric*(F, A,?)**** > > MatSolve(F,b,x)**** > > **2. **I have three dense matrices A at three different dimensions: > N=75, 2028 and 21180. **** > > **3. **The calling sequence works for N=75 and 2028. But when > N=21180, the program hanged up when calling *MatLUFactorNumeric*(?). > Seemed to be a segmentation fault with the following error message: > If your matrices are sparse, i.e., using petsc default aij format, then superlu/mumps should work for N=21180. Start with small number of processors, e.g., 4, 16 etc. If superlu_dist hangs, does mumps also hang? ** ** > > Does anybody have similar experience on that? > Segmentation fault occurs for various reasons. Give a try to mumps first. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 15 10:51:10 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Aug 2012 10:51:10 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> Message-ID: Blaise, We are confused. Can you run both the bt and l2 with all those options and send ALL the output from each of the two runs. Thanks Barry On Aug 14, 2012, at 6:24 PM, Blaise Bourdin wrote: > Hi, > > > On Aug 14, 2012, at 5:58 PM, Barry Smith wrote: > >> >> Blaise, >> >> You can run with -snes_linesearch_monitor -info -ksp_monitor_true_residual -ksp_converged_reason >> >> >> to get much more information about what is happening and why the line search is failing. > Focussing on the relevant part of the output, I get the following for the first SNES step > > *** Using l2 > Residual norms for temp_ solve. > 0 KSP preconditioned resid norm 2.352873068990e+00 true resid norm 3.742138023215e+00 ||r(i)||/||b|| 1.000000000000e+00 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS > 1 KSP preconditioned resid norm 7.175926524783e-01 true resid norm 8.026926174904e-01 ||r(i)||/||b|| 2.145010719836e-01 > 2 KSP preconditioned resid norm 4.099791012407e-01 true resid norm 6.219898727406e-01 ||r(i)||/||b|| 1.662124349455e-01 > 3 KSP preconditioned resid norm 2.769612150659e-01 true resid norm 4.622335508644e-01 ||r(i)||/||b|| 1.235212458752e-01 > 4 KSP preconditioned resid norm 8.991164116822e-02 true resid norm 1.938840972976e-01 ||r(i)||/||b|| 5.181104921701e-02 > 5 KSP preconditioned resid norm 1.369794733551e-02 true resid norm 2.867541652138e-02 ||r(i)||/||b|| 7.662843097578e-03 > 6 KSP preconditioned resid norm 3.522245138600e-03 true resid norm 5.452585588775e-03 ||r(i)||/||b|| 1.457077626466e-03 > 7 KSP preconditioned resid norm 1.117008651636e-03 true resid norm 1.551905826961e-03 ||r(i)||/||b|| 4.147110067382e-04 > 8 KSP preconditioned resid norm 5.008635361807e-04 true resid norm 6.226120116381e-04 ||r(i)||/||b|| 1.663786872038e-04 > 9 KSP preconditioned resid norm 2.079118910844e-04 true resid norm 3.430664466007e-04 ||r(i)||/||b|| 9.167658821571e-05 > 10 KSP preconditioned resid norm 4.528126074206e-05 true resid norm 9.520866575330e-05 ||r(i)||/||b|| 2.544231804457e-05 > 11 KSP preconditioned resid norm 8.441137224072e-06 true resid norm 1.519916221879e-05 ||r(i)||/||b|| 4.061625232553e-06 > 12 KSP preconditioned resid norm 1.839317974157e-06 true resid norm 3.245208227421e-06 ||r(i)||/||b|| 8.672069836252e-07 > 13 KSP preconditioned resid norm 4.346353491153e-07 true resid norm 7.198101954057e-07 ||r(i)||/||b|| 1.923526580100e-07 > 14 KSP preconditioned resid norm 6.321413805477e-08 true resid norm 1.280486229700e-07 ||r(i)||/||b|| 3.421803850515e-08 > 15 KSP preconditioned resid norm 9.029476674935e-09 true resid norm 2.193598397200e-08 ||r(i)||/||b|| 5.861885327562e-09 > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 9.029476674935e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 2.352873068990e+00 at iteration 15 > Linear solve converged due to CONVERGED_RTOL iterations 15 > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 4.179425981164e+00 near zero implies inconsistent rhs > Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] > Line search terminated: lambda = 1, fnorms = 2.19338e-08 > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=2.1933796240714327e-08, ynorm=5.1838420564550498e+00, lssucceed=1 > 1 SNES Function norm 2.193379624071e-08 > [0] SNESDefaultConverged(): Converged due to function norm 2.193379624071e-08 < 3.742138023215e-08 (relative tolerance) > SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 > > *** Using bt > Residual norms for temp_ solve. > 0 KSP preconditioned resid norm 1.176436534193e+00 true resid norm 3.742138022596e+00 ||r(i)||/||b|| 9.999999998344e-01 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS > 1 KSP preconditioned resid norm 3.587963259712e-01 true resid norm 8.026926179905e-01 ||r(i)||/||b|| 2.145010721173e-01 > 2 KSP preconditioned resid norm 2.049895509618e-01 true resid norm 6.219898720314e-01 ||r(i)||/||b|| 1.662124347560e-01 > 3 KSP preconditioned resid norm 1.384806072424e-01 true resid norm 4.622335514699e-01 ||r(i)||/||b|| 1.235212460370e-01 > 4 KSP preconditioned resid norm 4.495582078268e-02 true resid norm 1.938840967382e-01 ||r(i)||/||b|| 5.181104906751e-02 > 5 KSP preconditioned resid norm 6.848973644691e-03 true resid norm 2.867541656135e-02 ||r(i)||/||b|| 7.662843108259e-03 > 6 KSP preconditioned resid norm 1.761122593261e-03 true resid norm 5.452585577786e-03 ||r(i)||/||b|| 1.457077623530e-03 > 7 KSP preconditioned resid norm 5.585043310756e-04 true resid norm 1.551905808041e-03 ||r(i)||/||b|| 4.147110016822e-04 > 8 KSP preconditioned resid norm 2.504317746421e-04 true resid norm 6.226120013063e-04 ||r(i)||/||b|| 1.663786844429e-04 > 9 KSP preconditioned resid norm 1.039559493091e-04 true resid norm 3.430664433173e-04 ||r(i)||/||b|| 9.167658733830e-05 > 10 KSP preconditioned resid norm 2.264063167431e-05 true resid norm 9.520867156897e-05 ||r(i)||/||b|| 2.544231959867e-05 > 11 KSP preconditioned resid norm 4.220568874641e-06 true resid norm 1.519916304124e-05 ||r(i)||/||b|| 4.061625452335e-06 > 12 KSP preconditioned resid norm 9.196589910150e-07 true resid norm 3.245208482213e-06 ||r(i)||/||b|| 8.672070517123e-07 > 13 KSP preconditioned resid norm 2.173176852660e-07 true resid norm 7.198102176806e-07 ||r(i)||/||b|| 1.923526639624e-07 > 14 KSP preconditioned resid norm 3.160707194729e-08 true resid norm 1.280486368595e-07 ||r(i)||/||b|| 3.421804221680e-08 > 15 KSP preconditioned resid norm 4.514738683754e-09 true resid norm 2.193598711826e-08 ||r(i)||/||b|| 5.861886168328e-09 > [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.514738683754e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 1.176436534495e+00 at iteration 15 > Linear solve converged due to CONVERGED_RTOL iterations 15 > [0] SNESSolve_LS(): iter=0, linear solve iterations=15 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs > [0] VecScatterCreate(): Special case: sequential vector general scatter > [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 > [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 > 1 SNES Function norm 1.871069011453e+00 > > > As expected, the KSP residuals are exactly the same. I am not sure what to make of > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs. > Which RHS is this referring to? the RHS for SNESSolve is 0 (second argument of SNESSolve is PETSC_NULL_OBJECT). Could this mean that there is an incompatibility between my Jacobian and my Function? > > In all case that diverge, it looks like the gnorm in the linesearch does not decrease. > >> >> If that doesn't help you can send the code and we can play with it. > > The code is a bit of a pain to build but both Matt and Jed have accounts on my systems. I can arrange to give access to others if necessary. > > Blaise > > >> >> Barry >> >> >> >> On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: >> >>> HI, >>> >>> I am trying to understand if the following behavior is normal / expected: >>> >>> I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. >>> >>> In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. >>> >>> The relevant code is as follow: >>> Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) >>> Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) >>> Call VecCopy(solTemp,tmpvec,ierr) >>> >>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>> >>> Call VecCopy(tmpvec,soltemp,ierr) >>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>> >>> >>> Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. >>> >>> Blaise >>> >>> >>> >>> Below is the sample output using cp as the linesearch type in which case I get the expected convergence: >>> Solving time step 1: t= 1.00000E+00 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search terminated: lambda = 1, fnorms = 2.1936e-08 >>> 1 SNES Function norm 2.193598339906e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=3 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: cp >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=1 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] >>> Line search terminated: lambda = 2, fnorms = 2.1936e-08 >>> 1 SNES Function norm 2.193598717772e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=3 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: cp >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=1 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> >>> >>> when using the default linesearch (bt), the second SNESSolve fails: >>> >>> Solving time step 1: t= 1.00000E+00 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 >>> 1 SNES Function norm 2.193598339906e-08 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=15 >>> total number of function evaluations=2 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: bt >>> interpolation: cubic >>> alpha=1.000000e-04 >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=40 >>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>> 0 SNES Function norm 3.742138023215e+00 >>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >>> 1 SNES Function norm 1.871069011453e+00 >>> Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 >>> 2 SNES Function norm 1.247379340865e+00 >>> Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 >>> 3 SNES Function norm 9.355345056231e-01 >>> Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 >>> 4 SNES Function norm 7.484276044882e-01 >>> Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 >>> 5 SNES Function norm 6.236896704016e-01 >>> Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 >>> 6 SNES Function norm 5.345911460556e-01 >>> Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 >>> 7 SNES Function norm 4.677672528597e-01 >>> Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 >>> 8 SNES Function norm 4.157931136937e-01 >>> Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 >>> 9 SNES Function norm 3.742138023528e-01 >>> Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 >>> 10 SNES Function norm 3.401943657960e-01 >>> Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 >>> 11 SNES Function norm 3.118448353285e-01 >>> Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 >>> 12 SNES Function norm 2.878567710844e-01 >>> Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 >>> 13 SNES Function norm 2.672955731592e-01 >>> Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 >>> 14 SNES Function norm 2.494758682895e-01 >>> Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 >>> 15 SNES Function norm 2.338836265275e-01 >>> Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 >>> 16 SNES Function norm 2.201257661485e-01 >>> Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 >>> 17 SNES Function norm 2.078965569222e-01 >>> Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 >>> 18 SNES Function norm 1.969546328772e-01 >>> Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 >>> 19 SNES Function norm 1.871069012364e-01 >>> Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 >>> 20 SNES Function norm 1.781970487991e-01 >>> Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 >>> 21 SNES Function norm 1.700971829468e-01 >>> Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 >>> 22 SNES Function norm 1.627016532554e-01 >>> Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 >>> 23 SNES Function norm 1.559224177048e-01 >>> Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 >>> 24 SNES Function norm 1.496855209981e-01 >>> Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 >>> 25 SNES Function norm 1.439283855764e-01 >>> Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 >>> 26 SNES Function norm 1.385977046303e-01 >>> Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 >>> 27 SNES Function norm 1.336477866088e-01 >>> Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 >>> 28 SNES Function norm 1.290392422439e-01 >>> Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 >>> 29 SNES Function norm 1.247379341700e-01 >>> Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 >>> 30 SNES Function norm 1.207141298427e-01 >>> Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 >>> 31 SNES Function norm 1.169418132858e-01 >>> Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 >>> 32 SNES Function norm 1.133981219747e-01 >>> Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 >>> 33 SNES Function norm 1.100628830937e-01 >>> Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 >>> 34 SNES Function norm 1.069182292915e-01 >>> Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 >>> 35 SNES Function norm 1.039482784783e-01 >>> Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 >>> 36 SNES Function norm 1.011388655469e-01 >>> Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 >>> 37 SNES Function norm 9.847731645400e-02 >>> Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 >>> 38 SNES Function norm 9.595225705796e-02 >>> Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 >>> 39 SNES Function norm 9.355345063171e-02 >>> Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 >>> 40 SNES Function norm 9.127165915308e-02 >>> Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 >>> 41 SNES Function norm 8.909852441151e-02 >>> Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 >>> 42 SNES Function norm 8.702646570443e-02 >>> Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 >>> 43 SNES Function norm 8.504859148402e-02 >>> Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 >>> 44 SNES Function norm 8.315862278451e-02 >>> Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 >>> 45 SNES Function norm 8.135082663716e-02 >>> Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 >>> 46 SNES Function norm 7.961995798542e-02 >>> Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 >>> 47 SNES Function norm 7.796120886084e-02 >>> Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 >>> 48 SNES Function norm 7.637016378216e-02 >>> Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 >>> 49 SNES Function norm 7.484276050661e-02 >>> Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 >>> 50 SNES Function norm 7.337525539874e-02 >>> SNES Object:(temp_) 1 MPI processes >>> type: ls >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=693 >>> total number of function evaluations=51 >>> KSP Object: (temp_) 1 MPI processes >>> type: cg >>> maximum iterations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using nonzero initial guess >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (temp_) 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Matrix Object: 1 MPI processes >>> type: seqsbaij >>> rows=104, cols=104 >>> package used to perform factorization: petsc >>> total: nonzeros=381, allocated nonzeros=381 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Matrix Object: 1 MPI processes >>> type: seqaij >>> rows=104, cols=104 >>> total: nonzeros=658, allocated nonzeros=658 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> SNESLineSearch Object: (temp_) 1 MPI processes >>> type: bt >>> interpolation: cubic >>> alpha=1.000000e-04 >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>> maximum iterations=40 >>> SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 >>> >>> >>> >>> >>> >>> -- >>> Department of Mathematics and Center for Computation & Technology >>> Louisiana State University, Baton Rouge, LA 70803, USA >>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >>> >>> >>> >>> >>> >>> >>> >> > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > > From bourdin at lsu.edu Wed Aug 15 12:13:14 2012 From: bourdin at lsu.edu (Blaise Bourdin) Date: Wed, 15 Aug 2012 12:13:14 -0500 Subject: [petsc-users] LineSearch question In-Reply-To: References: <8B9E591A-C2A5-4246-8616-99795375B2A1@lsu.edu> <8C2A47FB-9AC8-4581-9FED-C94ACC08A470@lsu.edu> Message-ID: <1FADCE2F-060E-4D37-8F4E-C8C46E66AE69@lsu.edu> Hi, I figured it out, and it was my fault... I was not zeroing out the matrix before assembling the Jacobian. Once I did this, every linesearch converges and each SNES converge in 1 iteration, as expected. Sorry for the confusion. Blaise On Aug 15, 2012, at 10:51 AM, Barry Smith wrote: > > Blaise, > > We are confused. Can you run both the bt and l2 with all those options and send ALL the output from each of the two runs. > > Thanks > > Barry > > > On Aug 14, 2012, at 6:24 PM, Blaise Bourdin wrote: > >> Hi, >> >> >> On Aug 14, 2012, at 5:58 PM, Barry Smith wrote: >> >>> >>> Blaise, >>> >>> You can run with -snes_linesearch_monitor -info -ksp_monitor_true_residual -ksp_converged_reason >>> >>> >>> to get much more information about what is happening and why the line search is failing. >> Focussing on the relevant part of the output, I get the following for the first SNES step >> >> *** Using l2 >> Residual norms for temp_ solve. >> 0 KSP preconditioned resid norm 2.352873068990e+00 true resid norm 3.742138023215e+00 ||r(i)||/||b|| 1.000000000000e+00 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS >> 1 KSP preconditioned resid norm 7.175926524783e-01 true resid norm 8.026926174904e-01 ||r(i)||/||b|| 2.145010719836e-01 >> 2 KSP preconditioned resid norm 4.099791012407e-01 true resid norm 6.219898727406e-01 ||r(i)||/||b|| 1.662124349455e-01 >> 3 KSP preconditioned resid norm 2.769612150659e-01 true resid norm 4.622335508644e-01 ||r(i)||/||b|| 1.235212458752e-01 >> 4 KSP preconditioned resid norm 8.991164116822e-02 true resid norm 1.938840972976e-01 ||r(i)||/||b|| 5.181104921701e-02 >> 5 KSP preconditioned resid norm 1.369794733551e-02 true resid norm 2.867541652138e-02 ||r(i)||/||b|| 7.662843097578e-03 >> 6 KSP preconditioned resid norm 3.522245138600e-03 true resid norm 5.452585588775e-03 ||r(i)||/||b|| 1.457077626466e-03 >> 7 KSP preconditioned resid norm 1.117008651636e-03 true resid norm 1.551905826961e-03 ||r(i)||/||b|| 4.147110067382e-04 >> 8 KSP preconditioned resid norm 5.008635361807e-04 true resid norm 6.226120116381e-04 ||r(i)||/||b|| 1.663786872038e-04 >> 9 KSP preconditioned resid norm 2.079118910844e-04 true resid norm 3.430664466007e-04 ||r(i)||/||b|| 9.167658821571e-05 >> 10 KSP preconditioned resid norm 4.528126074206e-05 true resid norm 9.520866575330e-05 ||r(i)||/||b|| 2.544231804457e-05 >> 11 KSP preconditioned resid norm 8.441137224072e-06 true resid norm 1.519916221879e-05 ||r(i)||/||b|| 4.061625232553e-06 >> 12 KSP preconditioned resid norm 1.839317974157e-06 true resid norm 3.245208227421e-06 ||r(i)||/||b|| 8.672069836252e-07 >> 13 KSP preconditioned resid norm 4.346353491153e-07 true resid norm 7.198101954057e-07 ||r(i)||/||b|| 1.923526580100e-07 >> 14 KSP preconditioned resid norm 6.321413805477e-08 true resid norm 1.280486229700e-07 ||r(i)||/||b|| 3.421803850515e-08 >> 15 KSP preconditioned resid norm 9.029476674935e-09 true resid norm 2.193598397200e-08 ||r(i)||/||b|| 5.861885327562e-09 >> [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 9.029476674935e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 2.352873068990e+00 at iteration 15 >> Linear solve converged due to CONVERGED_RTOL iterations 15 >> [0] SNESSolve_LS(): iter=0, linear solve iterations=15 >> [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 4.179425981164e+00 near zero implies inconsistent rhs >> Line search: lambdas = [1, 0.5, 0], fnorms = [2.1936e-08, 1.87107, 3.74214] >> Line search terminated: lambda = 1, fnorms = 2.19338e-08 >> [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=2.1933796240714327e-08, ynorm=5.1838420564550498e+00, lssucceed=1 >> 1 SNES Function norm 2.193379624071e-08 >> [0] SNESDefaultConverged(): Converged due to function norm 2.193379624071e-08 < 3.742138023215e-08 (relative tolerance) >> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >> >> *** Using bt >> Residual norms for temp_ solve. >> 0 KSP preconditioned resid norm 1.176436534193e+00 true resid norm 3.742138022596e+00 ||r(i)||/||b|| 9.999999998344e-01 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS >> 1 KSP preconditioned resid norm 3.587963259712e-01 true resid norm 8.026926179905e-01 ||r(i)||/||b|| 2.145010721173e-01 >> 2 KSP preconditioned resid norm 2.049895509618e-01 true resid norm 6.219898720314e-01 ||r(i)||/||b|| 1.662124347560e-01 >> 3 KSP preconditioned resid norm 1.384806072424e-01 true resid norm 4.622335514699e-01 ||r(i)||/||b|| 1.235212460370e-01 >> 4 KSP preconditioned resid norm 4.495582078268e-02 true resid norm 1.938840967382e-01 ||r(i)||/||b|| 5.181104906751e-02 >> 5 KSP preconditioned resid norm 6.848973644691e-03 true resid norm 2.867541656135e-02 ||r(i)||/||b|| 7.662843108259e-03 >> 6 KSP preconditioned resid norm 1.761122593261e-03 true resid norm 5.452585577786e-03 ||r(i)||/||b|| 1.457077623530e-03 >> 7 KSP preconditioned resid norm 5.585043310756e-04 true resid norm 1.551905808041e-03 ||r(i)||/||b|| 4.147110016822e-04 >> 8 KSP preconditioned resid norm 2.504317746421e-04 true resid norm 6.226120013063e-04 ||r(i)||/||b|| 1.663786844429e-04 >> 9 KSP preconditioned resid norm 1.039559493091e-04 true resid norm 3.430664433173e-04 ||r(i)||/||b|| 9.167658733830e-05 >> 10 KSP preconditioned resid norm 2.264063167431e-05 true resid norm 9.520867156897e-05 ||r(i)||/||b|| 2.544231959867e-05 >> 11 KSP preconditioned resid norm 4.220568874641e-06 true resid norm 1.519916304124e-05 ||r(i)||/||b|| 4.061625452335e-06 >> 12 KSP preconditioned resid norm 9.196589910150e-07 true resid norm 3.245208482213e-06 ||r(i)||/||b|| 8.672070517123e-07 >> 13 KSP preconditioned resid norm 2.173176852660e-07 true resid norm 7.198102176806e-07 ||r(i)||/||b|| 1.923526639624e-07 >> 14 KSP preconditioned resid norm 3.160707194729e-08 true resid norm 1.280486368595e-07 ||r(i)||/||b|| 3.421804221680e-08 >> 15 KSP preconditioned resid norm 4.514738683754e-09 true resid norm 2.193598711826e-08 ||r(i)||/||b|| 5.861886168328e-09 >> [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.514738683754e-09 is less than relative tolerance 1.000000000000e-08 times initial right hand side norm 1.176436534495e+00 at iteration 15 >> Linear solve converged due to CONVERGED_RTOL iterations 15 >> [0] SNESSolve_LS(): iter=0, linear solve iterations=15 >> [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs >> [0] VecScatterCreate(): Special case: sequential vector general scatter >> [0] SNESLineSearchApply_BT(): Initial fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >> [0] SNESSolve_LS(): fnorm=3.7421380232151638e+00, gnorm=1.8710690114527022e+00, ynorm=2.5919210284812890e+00, lssucceed=1 >> 1 SNES Function norm 1.871069011453e+00 >> >> >> As expected, the KSP residuals are exactly the same. I am not sure what to make of >> [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 8.358851664967e+00 near zero implies inconsistent rhs. >> Which RHS is this referring to? the RHS for SNESSolve is 0 (second argument of SNESSolve is PETSC_NULL_OBJECT). Could this mean that there is an incompatibility between my Jacobian and my Function? >> >> In all case that diverge, it looks like the gnorm in the linesearch does not decrease. >> >>> >>> If that doesn't help you can send the code and we can play with it. >> >> The code is a bit of a pain to build but both Matt and Jed have accounts on my systems. I can arrange to give access to others if necessary. >> >> Blaise >> >> >>> >>> Barry >>> >>> >>> >>> On Aug 14, 2012, at 5:53 PM, Blaise Bourdin wrote: >>> >>>> HI, >>>> >>>> I am trying to understand if the following behavior is normal / expected: >>>> >>>> I am solving a quasi-static evolution where at each time step, SNESSolve is called. My validation problem is a _static_ problem with 2 time steps (i.e. 2 successive calls to SNESSolve with the same operator, jacobian, and right hand side). Furthermore, the problem is linear, so that the Jacobian is constant. I even reset the solution vector to the same value at each time step. >>>> >>>> In this setting, I am expecting that at each time step, each SNESSolve should converge in exactly one iteration no matter what linesearch / snes type I use. Indeed, when setting the linesearch to l2 or cp, this is what I get. However, for all other choices, the second call to SNESSolve fails to converge with a SNESConvergedReason of -5 or -6. >>>> >>>> The relevant code is as follow: >>>> Call VecSet(solTemp,0.0_Kr,ierr);CHKERRQ(ierr) >>>> Call FormInitialGuess(snesTemp,solTemp,MEF90Ctx,ierr);CHKERRQ(ierr) >>>> Call VecCopy(solTemp,tmpvec,ierr) >>>> >>>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>>> >>>> Call VecCopy(tmpvec,soltemp,ierr) >>>> Call SNESSolve(snesTemp,PETSC_NULL_OBJECT,solTemp,ierr);CHKERRQ(ierr) >>>> >>>> >>>> Is this expected? I tried to call SNESLineSearchReset before the second call to SNESSolve, but this does not seem to have any effect. >>>> >>>> Blaise >>>> >>>> >>>> >>>> Below is the sample output using cp as the linesearch type in which case I get the expected convergence: >>>> Solving time step 1: t= 1.00000E+00 >>>> 0 SNES Function norm 3.742138023215e+00 >>>> Line search terminated: lambda = 1, fnorms = 2.1936e-08 >>>> 1 SNES Function norm 2.193598339906e-08 >>>> SNES Object:(temp_) 1 MPI processes >>>> type: ls >>>> maximum iterations=50, maximum function evaluations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>>> total number of linear solver iterations=15 >>>> total number of function evaluations=3 >>>> KSP Object: (temp_) 1 MPI processes >>>> type: cg >>>> maximum iterations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using nonzero initial guess >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: (temp_) 1 MPI processes >>>> type: icc >>>> 0 levels of fill >>>> tolerance for zero pivot 2.22045e-14 >>>> using Manteuffel shift >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Matrix Object: 1 MPI processes >>>> type: seqsbaij >>>> rows=104, cols=104 >>>> package used to perform factorization: petsc >>>> total: nonzeros=381, allocated nonzeros=381 >>>> total number of mallocs used during MatSetValues calls =0 >>>> block size is 1 >>>> linear system matrix = precond matrix: >>>> Matrix Object: 1 MPI processes >>>> type: seqaij >>>> rows=104, cols=104 >>>> total: nonzeros=658, allocated nonzeros=658 >>>> total number of mallocs used during MatSetValues calls =0 >>>> not using I-node routines >>>> SNESLineSearch Object: (temp_) 1 MPI processes >>>> type: cp >>>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>>> maximum iterations=1 >>>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>>> 0 SNES Function norm 3.742138023215e+00 >>>> Line search: lambdas = [1, 0], ftys = [2.42597, 4.85193] >>>> Line search terminated: lambda = 2, fnorms = 2.1936e-08 >>>> 1 SNES Function norm 2.193598717772e-08 >>>> SNES Object:(temp_) 1 MPI processes >>>> type: ls >>>> maximum iterations=50, maximum function evaluations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>>> total number of linear solver iterations=15 >>>> total number of function evaluations=3 >>>> KSP Object: (temp_) 1 MPI processes >>>> type: cg >>>> maximum iterations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using nonzero initial guess >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: (temp_) 1 MPI processes >>>> type: icc >>>> 0 levels of fill >>>> tolerance for zero pivot 2.22045e-14 >>>> using Manteuffel shift >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Matrix Object: 1 MPI processes >>>> type: seqsbaij >>>> rows=104, cols=104 >>>> package used to perform factorization: petsc >>>> total: nonzeros=381, allocated nonzeros=381 >>>> total number of mallocs used during MatSetValues calls =0 >>>> block size is 1 >>>> linear system matrix = precond matrix: >>>> Matrix Object: 1 MPI processes >>>> type: seqaij >>>> rows=104, cols=104 >>>> total: nonzeros=658, allocated nonzeros=658 >>>> total number of mallocs used during MatSetValues calls =0 >>>> not using I-node routines >>>> SNESLineSearch Object: (temp_) 1 MPI processes >>>> type: cp >>>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>>> maximum iterations=1 >>>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>>> >>>> >>>> when using the default linesearch (bt), the second SNESSolve fails: >>>> >>>> Solving time step 1: t= 1.00000E+00 >>>> 0 SNES Function norm 3.742138023215e+00 >>>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 2.193598339906e-08 >>>> 1 SNES Function norm 2.193598339906e-08 >>>> SNES Object:(temp_) 1 MPI processes >>>> type: ls >>>> maximum iterations=50, maximum function evaluations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>>> total number of linear solver iterations=15 >>>> total number of function evaluations=2 >>>> KSP Object: (temp_) 1 MPI processes >>>> type: cg >>>> maximum iterations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using nonzero initial guess >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: (temp_) 1 MPI processes >>>> type: icc >>>> 0 levels of fill >>>> tolerance for zero pivot 2.22045e-14 >>>> using Manteuffel shift >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Matrix Object: 1 MPI processes >>>> type: seqsbaij >>>> rows=104, cols=104 >>>> package used to perform factorization: petsc >>>> total: nonzeros=381, allocated nonzeros=381 >>>> total number of mallocs used during MatSetValues calls =0 >>>> block size is 1 >>>> linear system matrix = precond matrix: >>>> Matrix Object: 1 MPI processes >>>> type: seqaij >>>> rows=104, cols=104 >>>> total: nonzeros=658, allocated nonzeros=658 >>>> total number of mallocs used during MatSetValues calls =0 >>>> not using I-node routines >>>> SNESLineSearch Object: (temp_) 1 MPI processes >>>> type: bt >>>> interpolation: cubic >>>> alpha=1.000000e-04 >>>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>>> maximum iterations=40 >>>> SNESTemp converged in in 1 iterations. SNESConvergedReason is 3 >>>> 0 SNES Function norm 3.742138023215e+00 >>>> Line search: Using full step: fnorm 3.742138023215e+00 gnorm 1.871069011453e+00 >>>> 1 SNES Function norm 1.871069011453e+00 >>>> Line search: Using full step: fnorm 1.871069011453e+00 gnorm 1.247379340865e+00 >>>> 2 SNES Function norm 1.247379340865e+00 >>>> Line search: Using full step: fnorm 1.247379340865e+00 gnorm 9.355345056231e-01 >>>> 3 SNES Function norm 9.355345056231e-01 >>>> Line search: Using full step: fnorm 9.355345056231e-01 gnorm 7.484276044882e-01 >>>> 4 SNES Function norm 7.484276044882e-01 >>>> Line search: Using full step: fnorm 7.484276044882e-01 gnorm 6.236896704016e-01 >>>> 5 SNES Function norm 6.236896704016e-01 >>>> Line search: Using full step: fnorm 6.236896704016e-01 gnorm 5.345911460556e-01 >>>> 6 SNES Function norm 5.345911460556e-01 >>>> Line search: Using full step: fnorm 5.345911460556e-01 gnorm 4.677672528597e-01 >>>> 7 SNES Function norm 4.677672528597e-01 >>>> Line search: Using full step: fnorm 4.677672528597e-01 gnorm 4.157931136937e-01 >>>> 8 SNES Function norm 4.157931136937e-01 >>>> Line search: Using full step: fnorm 4.157931136937e-01 gnorm 3.742138023528e-01 >>>> 9 SNES Function norm 3.742138023528e-01 >>>> Line search: Using full step: fnorm 3.742138023528e-01 gnorm 3.401943657960e-01 >>>> 10 SNES Function norm 3.401943657960e-01 >>>> Line search: Using full step: fnorm 3.401943657960e-01 gnorm 3.118448353285e-01 >>>> 11 SNES Function norm 3.118448353285e-01 >>>> Line search: Using full step: fnorm 3.118448353285e-01 gnorm 2.878567710844e-01 >>>> 12 SNES Function norm 2.878567710844e-01 >>>> Line search: Using full step: fnorm 2.878567710844e-01 gnorm 2.672955731592e-01 >>>> 13 SNES Function norm 2.672955731592e-01 >>>> Line search: Using full step: fnorm 2.672955731592e-01 gnorm 2.494758682895e-01 >>>> 14 SNES Function norm 2.494758682895e-01 >>>> Line search: Using full step: fnorm 2.494758682895e-01 gnorm 2.338836265275e-01 >>>> 15 SNES Function norm 2.338836265275e-01 >>>> Line search: Using full step: fnorm 2.338836265275e-01 gnorm 2.201257661485e-01 >>>> 16 SNES Function norm 2.201257661485e-01 >>>> Line search: Using full step: fnorm 2.201257661485e-01 gnorm 2.078965569222e-01 >>>> 17 SNES Function norm 2.078965569222e-01 >>>> Line search: Using full step: fnorm 2.078965569222e-01 gnorm 1.969546328772e-01 >>>> 18 SNES Function norm 1.969546328772e-01 >>>> Line search: Using full step: fnorm 1.969546328772e-01 gnorm 1.871069012364e-01 >>>> 19 SNES Function norm 1.871069012364e-01 >>>> Line search: Using full step: fnorm 1.871069012364e-01 gnorm 1.781970487991e-01 >>>> 20 SNES Function norm 1.781970487991e-01 >>>> Line search: Using full step: fnorm 1.781970487991e-01 gnorm 1.700971829468e-01 >>>> 21 SNES Function norm 1.700971829468e-01 >>>> Line search: Using full step: fnorm 1.700971829468e-01 gnorm 1.627016532554e-01 >>>> 22 SNES Function norm 1.627016532554e-01 >>>> Line search: Using full step: fnorm 1.627016532554e-01 gnorm 1.559224177048e-01 >>>> 23 SNES Function norm 1.559224177048e-01 >>>> Line search: Using full step: fnorm 1.559224177048e-01 gnorm 1.496855209981e-01 >>>> 24 SNES Function norm 1.496855209981e-01 >>>> Line search: Using full step: fnorm 1.496855209981e-01 gnorm 1.439283855764e-01 >>>> 25 SNES Function norm 1.439283855764e-01 >>>> Line search: Using full step: fnorm 1.439283855764e-01 gnorm 1.385977046303e-01 >>>> 26 SNES Function norm 1.385977046303e-01 >>>> Line search: Using full step: fnorm 1.385977046303e-01 gnorm 1.336477866088e-01 >>>> 27 SNES Function norm 1.336477866088e-01 >>>> Line search: Using full step: fnorm 1.336477866088e-01 gnorm 1.290392422439e-01 >>>> 28 SNES Function norm 1.290392422439e-01 >>>> Line search: Using full step: fnorm 1.290392422439e-01 gnorm 1.247379341700e-01 >>>> 29 SNES Function norm 1.247379341700e-01 >>>> Line search: Using full step: fnorm 1.247379341700e-01 gnorm 1.207141298427e-01 >>>> 30 SNES Function norm 1.207141298427e-01 >>>> Line search: Using full step: fnorm 1.207141298427e-01 gnorm 1.169418132858e-01 >>>> 31 SNES Function norm 1.169418132858e-01 >>>> Line search: Using full step: fnorm 1.169418132858e-01 gnorm 1.133981219747e-01 >>>> 32 SNES Function norm 1.133981219747e-01 >>>> Line search: Using full step: fnorm 1.133981219747e-01 gnorm 1.100628830937e-01 >>>> 33 SNES Function norm 1.100628830937e-01 >>>> Line search: Using full step: fnorm 1.100628830937e-01 gnorm 1.069182292915e-01 >>>> 34 SNES Function norm 1.069182292915e-01 >>>> Line search: Using full step: fnorm 1.069182292915e-01 gnorm 1.039482784783e-01 >>>> 35 SNES Function norm 1.039482784783e-01 >>>> Line search: Using full step: fnorm 1.039482784783e-01 gnorm 1.011388655469e-01 >>>> 36 SNES Function norm 1.011388655469e-01 >>>> Line search: Using full step: fnorm 1.011388655469e-01 gnorm 9.847731645400e-02 >>>> 37 SNES Function norm 9.847731645400e-02 >>>> Line search: Using full step: fnorm 9.847731645400e-02 gnorm 9.595225705796e-02 >>>> 38 SNES Function norm 9.595225705796e-02 >>>> Line search: Using full step: fnorm 9.595225705796e-02 gnorm 9.355345063171e-02 >>>> 39 SNES Function norm 9.355345063171e-02 >>>> Line search: Using full step: fnorm 9.355345063171e-02 gnorm 9.127165915308e-02 >>>> 40 SNES Function norm 9.127165915308e-02 >>>> Line search: Using full step: fnorm 9.127165915308e-02 gnorm 8.909852441151e-02 >>>> 41 SNES Function norm 8.909852441151e-02 >>>> Line search: Using full step: fnorm 8.909852441151e-02 gnorm 8.702646570443e-02 >>>> 42 SNES Function norm 8.702646570443e-02 >>>> Line search: Using full step: fnorm 8.702646570443e-02 gnorm 8.504859148402e-02 >>>> 43 SNES Function norm 8.504859148402e-02 >>>> Line search: Using full step: fnorm 8.504859148402e-02 gnorm 8.315862278451e-02 >>>> 44 SNES Function norm 8.315862278451e-02 >>>> Line search: Using full step: fnorm 8.315862278451e-02 gnorm 8.135082663716e-02 >>>> 45 SNES Function norm 8.135082663716e-02 >>>> Line search: Using full step: fnorm 8.135082663716e-02 gnorm 7.961995798542e-02 >>>> 46 SNES Function norm 7.961995798542e-02 >>>> Line search: Using full step: fnorm 7.961995798542e-02 gnorm 7.796120886084e-02 >>>> 47 SNES Function norm 7.796120886084e-02 >>>> Line search: Using full step: fnorm 7.796120886084e-02 gnorm 7.637016378216e-02 >>>> 48 SNES Function norm 7.637016378216e-02 >>>> Line search: Using full step: fnorm 7.637016378216e-02 gnorm 7.484276050661e-02 >>>> 49 SNES Function norm 7.484276050661e-02 >>>> Line search: Using full step: fnorm 7.484276050661e-02 gnorm 7.337525539874e-02 >>>> 50 SNES Function norm 7.337525539874e-02 >>>> SNES Object:(temp_) 1 MPI processes >>>> type: ls >>>> maximum iterations=50, maximum function evaluations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>>> total number of linear solver iterations=693 >>>> total number of function evaluations=51 >>>> KSP Object: (temp_) 1 MPI processes >>>> type: cg >>>> maximum iterations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> using nonzero initial guess >>>> using PRECONDITIONED norm type for convergence test >>>> PC Object: (temp_) 1 MPI processes >>>> type: icc >>>> 0 levels of fill >>>> tolerance for zero pivot 2.22045e-14 >>>> using Manteuffel shift >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Matrix Object: 1 MPI processes >>>> type: seqsbaij >>>> rows=104, cols=104 >>>> package used to perform factorization: petsc >>>> total: nonzeros=381, allocated nonzeros=381 >>>> total number of mallocs used during MatSetValues calls =0 >>>> block size is 1 >>>> linear system matrix = precond matrix: >>>> Matrix Object: 1 MPI processes >>>> type: seqaij >>>> rows=104, cols=104 >>>> total: nonzeros=658, allocated nonzeros=658 >>>> total number of mallocs used during MatSetValues calls =0 >>>> not using I-node routines >>>> SNESLineSearch Object: (temp_) 1 MPI processes >>>> type: bt >>>> interpolation: cubic >>>> alpha=1.000000e-04 >>>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 >>>> maximum iterations=40 >>>> SNESTemp converged in in 50 iterations. SNESConvergedReason is -5 >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Department of Mathematics and Center for Computation & Technology >>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >> >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> > -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From sean.null at gmail.com Wed Aug 15 12:13:40 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 15 Aug 2012 18:13:40 +0100 Subject: [petsc-users] Is there any counterpart of TSSetMatrices in petsc4py? Message-ID: Dear all, What I need is just to solve a very easy linear ODE. u_t = Au A is sparse and constant. But I didn't find a counterpart of TSSetMatrices in petsc4py for this. It is strange because petsc4py can handle much harder problems. Best, Xin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Aug 15 12:45:06 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 15 Aug 2012 14:45:06 -0300 Subject: [petsc-users] Is there any counterpart of TSSetMatrices in petsc4py? In-Reply-To: References: Message-ID: On 15 August 2012 14:13, Xin Zhao wrote: > Dear all, > > What I need is just to solve a very easy linear ODE. > > u_t = Au > > A is sparse and constant. > > But I didn't find a counterpart of TSSetMatrices in petsc4py for this. > It is strange because petsc4py can handle much harder problems. > TSSetMatrices() is no longer supported in petsc-3.3. You have to rewrite your problem in implicit nonlinear form, and ask TS to solve a linear problem. You still have to code your IFunction and IJacobian in Python, use the PETSc C implementations TSComputeIFunctionLinear() and TSComputeIJacobianConstant() and translate to Python. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From sean.null at gmail.com Wed Aug 15 13:37:10 2012 From: sean.null at gmail.com (Xin Zhao) Date: Wed, 15 Aug 2012 19:37:10 +0100 Subject: [petsc-users] Is there any counterpart of TSSetMatrices in petsc4py? In-Reply-To: References: Message-ID: Thanks. Best, Xin On Wed, Aug 15, 2012 at 6:45 PM, Lisandro Dalcin wrote: > On 15 August 2012 14:13, Xin Zhao wrote: > > Dear all, > > > > What I need is just to solve a very easy linear ODE. > > > > u_t = Au > > > > A is sparse and constant. > > > > But I didn't find a counterpart of TSSetMatrices in petsc4py for this. > > It is strange because petsc4py can handle much harder problems. > > > > TSSetMatrices() is no longer supported in petsc-3.3. You have to > rewrite your problem in implicit nonlinear form, and ask TS to solve a > linear problem. You still have to code your IFunction and IJacobian in > Python, use the PETSc C implementations TSComputeIFunctionLinear() and > TSComputeIJacobianConstant() and translate to Python. > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Aug 16 11:14:55 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 16 Aug 2012 16:14:55 +0000 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <500F4234-3C1C-4F81-9AF5-3F30EC4D0A24@mcs.anl.gov> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> <7237384C7A8F2642A8854B751619FA316F2738A5@exchange.scsolutions.com> <500F4234-3C1C-4F81-9AF5-3F30EC4D0A24@mcs.anl.gov> Message-ID: <7237384C7A8F2642A8854B751619FA316F273A78@exchange.scsolutions.com> Barry, ********************************************************************************************************************************* Here is the typical info I got from Valgrind for a large matrix with order N=141081. Would they imply something wrong in the installation or the software setting? ********************************************************************************************************************************* 1. Several circumstances where the following (Conditional jump or move) happens: ==28565== Conditional jump or move depends on uninitialised value(s) ==28565== at 0x85A9590: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE4066AA: ibv_open_device (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79BD990: ring_rdma_open_hca (rdma_iba_priv.c:411) ==28565== by 0x79CC6CD: rdma_setup_startup_ring (ring_startup.c:405) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) ==28565== Conditional jump or move depends on uninitialised value(s) ==28565== at 0x79A59D1: calloc (mvapich_malloc.c:3719) ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) ==28565== Conditional jump or move depends on uninitialised value(s) ==28565== at 0x4A08E54: memset (mc_replace_strmem.c:731) ==28565== by 0x79A5BA1: calloc (mvapich_malloc.c:3825) ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) ==28565== Conditional jump or move depends on uninitialised value(s) ==28565== at 0x4A08E79: memset (mc_replace_strmem.c:731) ==28565== by 0x79A5BA1: calloc (mvapich_malloc.c:3825) ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) 2. ==28565== Syscall param write(buf) points to uninitialised byte(s) ==28565== at 0x3FDF40E460: __write_nocancel (in /lib64/libpthread-2.12.so) ==28565== by 0x3FDE404DFC: ibv_cmd_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x85AB742: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) ==28565== Address 0x7feffbae8 is on thread 1's stack 3. ==28565== Use of uninitialised value of size 8 ==28565== at 0x85A998F: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) ==28565== by 0x408ED9: main (hw.cpp:122) ******************************************* The error message for rank 10 from PETSc is ******************************************* [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [10]PETSC ERROR: INSTEAD the line number of the start of the function [10]PETSC ERROR: is given. [10]PETSC ERROR: --------------------- Error Message ------------------------------------ [10]PETSC ERROR: Signal received! [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [10]PETSC ERROR: See docs/changes/index.html for recent updates. [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [10]PETSC ERROR: See docs/index.html for manual pages. [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: /nfs/06/com0488/programs/examples/testc/ZSOL on a arch-linu named n0685.ten.osc.edu by com0488 Wed Aug 15 21:00:31 2012 [10]PETSC ERROR: Libraries linked from /nfs/07/com0489/petsc/petsc-3.3-p2/arch-linux2-cxx-debug/lib [10]PETSC ERROR: Configure run at Wed Aug 15 13:55:29 2012 [10]PETSC ERROR: Configure options --with-blas-lapack-lib="-L/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -liomp5 -lm" --download-blacs --download-scalapack --with-mpi-dir=/usr/local/mvapich2/1.7-gnu --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic --download-mumps --download-superlu_dist --download-parmetis --download-metis --with-fortran-interfaces [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [cli_10]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Tuesday, August 14, 2012 7:54 PM To: PETSc users list Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() On Aug 14, 2012, at 6:05 PM, Jinquan Zhong wrote: > Barry, > > The machine I ran this program does not have valgrind. > > Another interesting observation is that when I ran the same three > matrices using PETSc3.2. MatLUFactorNumeric() hanged up even on N=75, > 2028 till I specified -mat_superlu_dist_colperm. However, > MatLUFactorNumeric() didn't work for N=21180 either even I used > > -mat_superlu_dist_rowperm NATURAL -mat_superlu_dist_colperm NATURAL > -mat_superlu_dist_parsymbfact YES > > I suspect that there is something incompatible in the factored matrix from superLU-dist to be used MatLUFactorNumeric() in PETSc3.2. Although PETSc 3.3 fixed this issue for matrix with small N, however, this issue relapsed for large N in PETSc3.3. It is using Superlu_dist for this factorization (and that version changed with PETSc 3.3) the problem is with Superlu_Dist not PETSc. valgrind will likely find an error in SuperLU_dist Barry > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Tuesday, August 14, 2012 3:55 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > > Can you run with valgrind > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: > >> Thanks, Matt. >> >> 1. Yes, I have checked the returned values from x obtained from >> MatSolve(F,b,x) >> >> The norm error check for x is complete for N=75, 2028. >> >> 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. >> >> >> [391]PETSC ERROR: >> --------------------------------------------------------------------- >> - >> -- [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >> Violation, probably memory access out of range [391]PETSC ERROR: Try >> option -start_in_debugger or -on_error_attach_debugger [391]PETSC >> ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETS >> C >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors [391]PETSC ERROR: likely location of >> problem given in stack below [391]PETSC ERROR: --------------------- >> Stack Frames ------------------------------------ >> [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [391]PETSC ERROR: INSTEAD the line number of the start of the function >> [391]PETSC ERROR: is given. >> [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/a >> i j/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] >> MatLUFactorNumeric line 2778 >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interfa >> c e/matrix.c [391]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [391]PETSC ERROR: Signal received! >> [391]PETSC ERROR: >> --------------------------------------------------------------------- >> - >> -- [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >> 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for >> recent updates. >> [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [391]PETSC ERROR: See docs/index.html for manual pages. >> [391]PETSC ERROR: >> --------------------------------------------------------------------- >> - >> -- [391]PETSC ERROR: >> /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL >> on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 >> 2012 [391]PETSC ERROR: Libraries linked from >> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx >> - debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 >> 2012 [391]PETSC ERROR: Configure options >> --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/li >> b >> /librefblas.a >> --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/ >> l ib/libreflapack.a --download-blacs --download-scalapack >> --with-mpi-dir=/usr/local/mvapich2/1.7-gnu >> --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex >> --with-precision=double --with-clanguage=cxx >> --with-fortran-kernels=generic --download-mumps >> --download-superlu_dist --download-parmetis --download-metis >> --with-fortran-interfaces[391]PETSC ERROR: >> --------------------------------------------------------------------- >> - >> -- [391]PETSC ERROR: User provided function() line 0 in unknown >> directory unknown file >> [cli_391]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 >> >> >> From: petsc-users-bounces at mcs.anl.gov >> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Tuesday, August 14, 2012 3:34 PM >> To: PETSc users list >> Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() >> >> On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: >> Dear PETSc folks, >> >> I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: >> >> 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is >> >> MatGetOrdering(A,...) >> >> MatGetFactor(A,...) >> >> MatLUFactorSymbolic(F, A,...) >> >> MatLUFactorNumeric(F, A,...) >> >> MatSolve(F,b,x) >> >> 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. >> >> 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: >> >> >> >> [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Signal received! >> >> ALWAYS send the entire error message. How can we tell anything from a small snippet? >> >> Since you have [1], this was run in parallel, so you need 3rd party >> packages. But you do not seem to be checking return values. Check >> them to make sure those packages are installed correctly. >> >> Matt >> >> Does anybody have similar experience on that? >> >> Thanks a lot! >> >> Jinquan >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From thomas.witkowski at tu-dresden.de Fri Aug 17 03:10:51 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Fri, 17 Aug 2012 10:10:51 +0200 Subject: [petsc-users] Nullspace for a coupled system of equations Message-ID: <502DFC8B.4080800@tu-dresden.de> I want to solve some (weakly) coupled system of equations of the following form: A B u . = ..... 0 C v so, C is the discrete Laplacian and A and B are some more complicated operators (I make use of linear finite elements). All boundary conditions are periodic, so the unknown v is determined only up to a constant. A and B contain both the identity operator, so u is fixed. Now I want to solve the system on the whole (there are reasons to do it in this way!) and I must provide information about the nullspace to the solver. When I am right, to provide the correct nullspace I must solve one equation with A. Is there any way in PETSc to circumvent the problem? Thomas From Vincent.De-Groof at uibk.ac.at Fri Aug 17 03:25:49 2012 From: Vincent.De-Groof at uibk.ac.at (De Groof, Vincent) Date: Fri, 17 Aug 2012 08:25:49 +0000 Subject: [petsc-users] Segmentation fault when using matlab interface Message-ID: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> Hello, I'm trying to use the Petsc objects directly from Matlab. However, when testing the examples in the /bin/matlab/classes/examples/tutorials-directory, I get a segmentation fault (core dump) error. Unfortunately, I don't get more information then this. After running some different situations, I know the following: I ran matlab both with and without desktop. The only error message I get is the segmentation fault. Sometimes it doesn't give the segmentation error after running the example. If so, it will cause a segmentation fault upon entering the next command in the matlab shell. If I quit matlab immediately after running an example without segmentation fault, my unix shell will hang. I know it's kind of vague, but this is all I know now. Any suggestions to look at to solve this problem? thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 06:43:31 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 05:43:31 -0600 Subject: [petsc-users] Segmentation fault when using matlab interface In-Reply-To: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> Message-ID: Can you set up a stand-alone test case that exhibits the error? On Fri, Aug 17, 2012 at 2:25 AM, De Groof, Vincent < Vincent.De-Groof at uibk.ac.at> wrote: > Hello, > > I'm trying to use the Petsc objects directly from Matlab. However, when > testing the examples in the > /bin/matlab/classes/examples/tutorials-directory, I get a segmentation > fault (core dump) error. Unfortunately, I don't get more information then > this. After running some different situations, I know the following: > I ran matlab both with and without desktop. The only error message I get > is the segmentation fault. Sometimes it doesn't give the segmentation error > after running the example. If so, it will cause a segmentation fault upon > entering the next command in the matlab shell. If I quit matlab immediately > after running an example without segmentation fault, my unix shell will > hang. > > > I know it's kind of vague, but this is all I know now. Any suggestions to > look at to solve this problem? > > > thanks. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 07:48:11 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 14:48:11 +0200 Subject: [petsc-users] Composite shell preconditiner Message-ID: <502E3D8B.3060100@gfz-potsdam.de> Hello petsc team! I would like to apply some sort of correction during Krylov process every N iterations. So I decided to use composite preconditioner like that: -pc_type composite -pc_composite_pcs shell,sor/icc/gamg I've started to test with the simplest shell preconditioner that does nothing: subroutine ShellPCApply(pc,x,y,ierr) implicit none #include "finclude/petscsys.h" #include "finclude/petscvec.h" PC,intent(in) :: pc Vec,intent(in) :: x Vec,intent(inout) :: y call VecCopy(x,y,ierr);CHKERRQ(ierr) endsubroutine ShellPCApply And it turned out that this influences convergence by itself. In case: -pc_type composite -pc_composite_pcs shell,sor ----> Linear solve converged due to CONVERGED_RTOL iterations 125 whereas -pc_type composite -pc_composite_pcs sor -----> Linear solve converged due to CONVERGED_RTOL iterations 35 Why does it happen? What is the right way to go then? Thanks in advance. Just in case, here is -ksp_view for -pc_composite_pcs shell,sor : PC Object:() 1 MPI processes type: composite Composite PC type - ADDITIVE PCs on composite preconditioner follow --------------------------------- PC Object: (sub_0_) 1 MPI processes type: shell Shell: no name linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=59049, cols=59049 total: nonzeros=741555, allocated nonzeros=767637 total number of mallocs used during MatSetValues calls =0 not using I-node routines PC Object: (sub_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=59049, cols=59049 total: nonzeros=741555, allocated nonzeros=767637 total number of mallocs used during MatSetValues calls =0 not using I-node routines --------------------------------- linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=59049, cols=59049 total: nonzeros=741555, allocated nonzeros=767637 total number of mallocs used during MatSetValues calls =0 not using I-node routines -- Regards, Alexander From jedbrown at mcs.anl.gov Fri Aug 17 07:52:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 06:52:23 -0600 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E3D8B.3060100@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> Message-ID: On Fri, Aug 17, 2012 at 6:48 AM, Alexander Grayver wrote: > Hello petsc team! > > I would like to apply some sort of correction during Krylov process every > N iterations. > So I decided to use composite preconditioner like that: > > -pc_type composite -pc_composite_pcs shell,sor/icc/gamg > > I've started to test with the simplest shell preconditioner that does > nothing: > > subroutine ShellPCApply(pc,x,y,ierr) > implicit none > > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > > PC,intent(in) :: pc > Vec,intent(in) :: x > Vec,intent(inout) :: y > > call VecCopy(x,y,ierr);CHKERRQ(**ierr) > endsubroutine ShellPCApply > > And it turned out that this influences convergence by itself. In case: > -pc_type composite -pc_composite_pcs shell,sor > ----> > Linear solve converged due to CONVERGED_RTOL iterations 125 > > whereas > -pc_type composite -pc_composite_pcs sor > -----> > Linear solve converged due to CONVERGED_RTOL iterations 35 > > Why does it happen? What is the right way to go then? > Thanks in advance. > > Just in case, here is -ksp_view for -pc_composite_pcs shell,sor : > > PC Object:() 1 MPI processes > type: composite > Composite PC type - ADDITIVE > See this *ADDITIVE*? You are preconditioning with I+P_{SOR}. You probably want multiplicative. > PCs on composite preconditioner follow > ------------------------------**--- > PC Object: (sub_0_) 1 MPI processes > type: shell > Shell: no name > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=59049, cols=59049 > total: nonzeros=741555, allocated nonzeros=767637 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > PC Object: (sub_1_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=59049, cols=59049 > total: nonzeros=741555, allocated nonzeros=767637 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > ------------------------------**--- > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=59049, cols=59049 > total: nonzeros=741555, allocated nonzeros=767637 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 08:11:45 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 15:11:45 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: References: <502E3D8B.3060100@gfz-potsdam.de> Message-ID: <502E4311.2090503@gfz-potsdam.de> On 17.08.2012 14:52, Jed Brown wrote: > > See this *ADDITIVE*? You are preconditioning with I+P_{SOR}. You > probably want multiplicative. Jed, It didn't work. -pc_type composite -pc_composite_pcs shell,sor -pc_composite_type multiplicative -----> Linear solve did not converge due to DIVERGED_ITS iterations 250 KSP Object:() 1 MPI processes type: tfqmr maximum iterations=250, initial guess is zero tolerances: relative=1e-07, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object:() 1 MPI processes type: composite Composite PC type - MULTIPLICATIVE PCs on composite preconditioner follow .... Is this something unexpected? I can try to write minimal example to make sure that this is not related to my app itself. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 08:30:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 07:30:41 -0600 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E4311.2090503@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> Message-ID: On Fri, Aug 17, 2012 at 7:11 AM, Alexander Grayver wrote: > On 17.08.2012 14:52, Jed Brown wrote: > > > See this *ADDITIVE*? You are preconditioning with I+P_{SOR}. You probably > want multiplicative. > > Jed, > > It didn't work. > > -pc_type composite -pc_composite_pcs shell,sor -pc_composite_type > multiplicative > -----> > Linear solve did not converge due to DIVERGED_ITS iterations 250 > > KSP Object:() 1 MPI processes > type: tfqmr > maximum iterations=250, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > > PC Object:() 1 MPI processes > type: composite > Composite PC type - MULTIPLICATIVE > > PCs on composite preconditioner follow > .... > > Is this something unexpected? > Sorry, I misinterpreted what you want. PC_COMPOSITE_MULTIPLICATIVE re-evaluates the residual after applying the first stage of the preconditioner. Do you _really_ want to just apply one preconditioner after the next? You can use PC_COMPOSITE_SPECIAL, but read this man page and understand why your preconditioner has that form. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCCompositeType.html Barry, it looks like despite the interface, "alpha" has never been used by this implementation. > I can try to write minimal example to make sure that this is not related > to my app itself. > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 08:36:23 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 15:36:23 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> Message-ID: <502E48D7.4070205@gfz-potsdam.de> On 17.08.2012 15:30, Jed Brown wrote: > > Sorry, I misinterpreted what you want. PC_COMPOSITE_MULTIPLICATIVE > re-evaluates the residual after applying the first stage of the > preconditioner. Do you _really_ want to just apply one preconditioner > after the next? You can use PC_COMPOSITE_SPECIAL, but read this man > page and understand why your preconditioner has that form. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCCompositeType.html Jed, I've seen this, but I'm not sure is it fits me. What I would like to do is very simple: every N iterations I want to take current solution vector *x*_i from KSP and calculate: *x*_i = *x*_i + *c*, where *c* is a correction vector. I want to be able to apply other preconditioners in a usual way. Is it possible? -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 08:45:06 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 07:45:06 -0600 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E48D7.4070205@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> Message-ID: On Fri, Aug 17, 2012 at 7:36 AM, Alexander Grayver wrote: > I've seen this, but I'm not sure is it fits me. > > What I would like to do is very simple: every N iterations I want to take > current solution vector *x*_i from KSP and calculate: > *x*_i = *x*_i + *c*, > where *c* is a correction vector. > How do you want to compute c? For a general KSP, this is not allowed because the preconditioner must be constant. (You can do it anyway by checking the iteration number in your PCShell, but it won't converge.) You can play these games with a flexible method like FGMRES, for which you can also do an inner-outer, e.g. -ksp_type fgmres -pc_type composite -pc_composite_type multiplicative -pc_composite_pcs ksp,shell -sub_0_ksp_ksp_type tfqmr -sub_0_ksp_ksp_max_it 10 -sub_0_ksp_pc_type ilu I want to be able to apply other preconditioners in a usual way. > > Is it possible? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 08:54:22 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 15:54:22 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> Message-ID: <502E4D0E.4080800@gfz-potsdam.de> On 17.08.2012 15:45, Jed Brown wrote: > > > What I would like to do is very simple: every N iterations I want > to take current solution vector *x*_i from KSP and calculate: > *x*_i = *x*_i + *c*, > where *c* is a correction vector. > > > How do you want to compute c? I was planning to do it inside ShellPCApply. The correction itself is calculated as a solution of Poisson equation with rhs being divergence of current density and current density is calculated based on *x*_i This trick is often applied to suppress spurious solution arising during solution of Maxwell equations and it works very well. > For a general KSP, this is not allowed because the preconditioner must > be constant. (You can do it anyway by checking the iteration number in > your PCShell, but it won't converge.) You can play these games with a > flexible method like FGMRES, for which you can also do an inner-outer, > e.g. > > -ksp_type fgmres -pc_type composite -pc_composite_type multiplicative > -pc_composite_pcs ksp,shell -sub_0_ksp_ksp_type tfqmr > -sub_0_ksp_ksp_max_it 10 -sub_0_ksp_pc_type ilu Looks a bit tricky. The number of fgmres iterations then defines number of the cycles correction will be applied, doesn't it? -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 09:03:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 09:03:19 -0500 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E4D0E.4080800@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> Message-ID: On Fri, Aug 17, 2012 at 8:54 AM, Alexander Grayver wrote: > I was planning to do it inside ShellPCApply. > The correction itself is calculated as a solution of Poisson equation with > rhs being divergence of current density and current density is calculated > based on *x*_i > Fine, sounds like you are doing divergence cleaning for an incompatible spatial discretization. Since the change lies within the null space of the intended continuous operator, you should be okay. But note that this procedure is fragile and will have different effect with different Krylov methods and preconditioning sides. > This trick is often applied to suppress spurious solution arising during > solution of Maxwell equations and it works very well. > > For a general KSP, this is not allowed because the preconditioner must > be constant. (You can do it anyway by checking the iteration number in your > PCShell, but it won't converge.) You can play these games with a flexible > method like FGMRES, for which you can also do an inner-outer, e.g. > > -ksp_type fgmres -pc_type composite -pc_composite_type multiplicative > -pc_composite_pcs ksp,shell -sub_0_ksp_ksp_type tfqmr -sub_0_ksp_ksp_max_it > 10 -sub_0_ksp_pc_type ilu > > > Looks a bit tricky. The number of fgmres iterations then defines number of > the cycles correction will be applied, doesn't it? > Yes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Aug 17 09:11:16 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 10:11:16 -0400 Subject: [petsc-users] GAMG setup scalability Message-ID: GAMG without any options is using MatTransposeMatMult (in petsc-dev) during setup at line 1031 of agg.c. What I find is that this operation takes up a majority of the total setup time, and doesn't scale well. Is there anything that can be done about this? I am a little surprised that it is taking significantly more time than the RAP construction of the coarse grid operators done by MatPtAP. On 1 processor, for example, it takes 5% of the setup time, and on 8 processors it takes ~4%. The MatTransposeMatMult, on the other hand, takes 67% of setup on 1 processor and 71% of setup on 8. I've attached the log_summary for 1 processor and 8 processors. You can also see that the solve time is completely eclipsed by the setup time. Regards, John -------------- next part -------------- A non-text attachment was scrubbed... Name: gamg_np1.out Type: application/octet-stream Size: 14732 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gamg_np8.out Type: application/octet-stream Size: 15496 bytes Desc: not available URL: From knepley at gmail.com Fri Aug 17 09:24:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 09:24:05 -0500 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: <502DFC8B.4080800@tu-dresden.de> References: <502DFC8B.4080800@tu-dresden.de> Message-ID: On Fri, Aug 17, 2012 at 3:10 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > I want to solve some (weakly) coupled system of equations of the following > form: > > A B u > . = ..... > 0 C v > > > so, C is the discrete Laplacian and A and B are some more complicated > operators (I make use of linear finite elements). All boundary conditions > are periodic, so the unknown v is determined only up to a constant. A and B > contain both the identity operator, so u is fixed. Now I want to solve the > system on the whole (there are reasons to do it in this way!) and I must > provide information about the nullspace to the solver. When I am right, to > provide the correct nullspace I must solve one equation with A. Is there > any way in PETSc to circumvent the problem? If I understand you correctly, your null space vector is (0 I). I use the same null space for SNES ex62. Note that you should probably use FieldSplit multiplicative here since you have a block triangular system to start with. Matt > > Thomas -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 09:23:54 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 16:23:54 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> Message-ID: <502E53FA.9070400@gfz-potsdam.de> On 17.08.2012 16:03, Jed Brown wrote: > > Fine, sounds like you are doing divergence cleaning for an > incompatible spatial discretization. Since the change lies within the > null space of the intended continuous operator, you should be okay. > But note that this procedure is fragile and will have different effect > with different Krylov methods and preconditioning sides. Right. Could you please explain what do you mean by fragile? > > Looks a bit tricky. The number of fgmres iterations then defines > number of the cycles correction will be applied, doesn't it? > > > Yes. If I do this command line only setup: -pc_type composite -pc_composite_type multiplicative -pc_composite_pcs ksp,shell How does petsc know what ShellPCApply routine to take? I guess I have to specify this in the code anyway? -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 09:32:07 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 09:32:07 -0500 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E53FA.9070400@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> <502E53FA.9070400@gfz-potsdam.de> Message-ID: On Fri, Aug 17, 2012 at 9:23 AM, Alexander Grayver wrote: > Right. > Could you please explain what do you mean by fragile? > Fragile means unreliable. The math is not right for Krylov methods in general. You are relying on side-effects of particular methods and preconditioner side combinations. Fragile is the opposite of "robust". For a "robust" method, you should start with a compatible discretization and/or choose a preconditioner that is stable on the quasi-null space and/or filter in a way that is consistent with the Krylov method (e.g. as described using FGMRES). > >> Looks a bit tricky. The number of fgmres iterations then defines number >> of the cycles correction will be applied, doesn't it? >> > > Yes. > > > If I do this command line only setup: > > -pc_type composite -pc_composite_type multiplicative -pc_composite_pcs > ksp,shell > > How does petsc know what ShellPCApply routine to take? I guess I have to > specify this in the code anyway? > Yeah, to select your preconditioner in this way, you should name it preconditioner something meaningful and use PCRegisterDynamic() just like the native implementations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Fri Aug 17 09:40:55 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 17 Aug 2012 07:40:55 -0700 Subject: [petsc-users] 64 bit integers Message-ID: <502E57F7.8020601@uci.edu> Hi, I want to build my application with 64 bit integer support because of the size of the problem I am working on. I compiled petsc with the option /--with-64-bit-indices=1/. In my Fortran code some petsc vectors are initialize by taking values from Fortran arrays. I used PetscInt indexes to reference both petsc and fortran vectors. It used to work with 32 bit indexes. Will it still be the case for the 64 bit case? Thank you / / -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 09:43:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 09:43:11 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E57F7.8020601@uci.edu> References: <502E57F7.8020601@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 9:40 AM, Michele Rosso wrote: > I want to build my application with 64 bit integer support > because of the size of the problem I am working on. > I compiled petsc with the option *--with-64-bit-indices=1*. > In my Fortran code some petsc vectors are initialize by taking > values from Fortran arrays. > I used PetscInt indexes to reference both petsc and fortran vectors. > It used to work with 32 bit indexes. > Will it still be the case for the 64 bit case? > As long as you use PetscInt for every integer that you pass to and from PETSc, it will work automatically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Aug 17 09:42:21 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Aug 2012 16:42:21 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> <502E53FA.9070400@gfz-potsdam.de> Message-ID: <502E584D.5050503@gfz-potsdam.de> Jed, I do see the drawbacks of this method, but it seems to have good scalability. That I want to test. Thank you. On 17.08.2012 16:32, Jed Brown wrote: > On Fri, Aug 17, 2012 at 9:23 AM, Alexander Grayver > > wrote: > > Right. > Could you please explain what do you mean by fragile? > > > Fragile means unreliable. The math is not right for Krylov methods in > general. You are relying on side-effects of particular methods and > preconditioner side combinations. Fragile is the opposite of "robust". > For a "robust" method, you should start with a compatible > discretization and/or choose a preconditioner that is stable on the > quasi-null space and/or filter in a way that is consistent with the > Krylov method (e.g. as described using FGMRES). > >> >> Looks a bit tricky. The number of fgmres iterations then >> defines number of the cycles correction will be applied, >> doesn't it? >> >> >> Yes. > > If I do this command line only setup: > > -pc_type composite -pc_composite_type multiplicative > -pc_composite_pcs ksp,shell > > How does petsc know what ShellPCApply routine to take? I guess I > have to specify this in the code anyway? > > > Yeah, to select your preconditioner in this way, you should name it > preconditioner something meaningful and use PCRegisterDynamic() just > like the native implementations. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Aug 17 09:50:36 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 17 Aug 2012 10:50:36 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: Message-ID: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> Yes John, the GAMG setup is not great. So we know there are problems. We do have a new staff person that will be working on matrix product methods soon. Matrix products are hard to make fast and GAMG setup relies on them quite a bit. A few things to note: 1) This is in the mesh setup phase, so many applications can amortize this cost. (MatPtAP is a "matrix" setup cost and so it is amortized for linear problems or non-full Newton solves, but not for full nonlinear solves). 2) You do not seem to be smoothing (-pc_gamg_agg_nsmooths 1), this is very useful for elliptic operators but uses a MatMatMult (so it gets worse). 3) This test is getting less than 2x speedup in KSPSolve for 8x processors. So this problem looks hard: small or poorly partitioned, and not in the range of where we want people to run to get good performance. 4) I have found that the setup times are about twice that of ML, which uses a similar algorithm, and about 5x slower than hypre, which uses a very different algorithm. So if you can not amortize theses setup costs then ML or hypre would probably work better for you. Mark On Aug 17, 2012, at 10:11 AM, John Fettig wrote: > GAMG without any options is using MatTransposeMatMult (in petsc-dev) > during setup at line 1031 of agg.c. What I find is that this > operation takes up a majority of the total setup time, and doesn't > scale well. Is there anything that can be done about this? > > I am a little surprised that it is taking significantly more time than > the RAP construction of the coarse grid operators done by MatPtAP. On > 1 processor, for example, it takes 5% of the setup time, and on 8 > processors it takes ~4%. The MatTransposeMatMult, on the other hand, > takes 67% of setup on 1 processor and 71% of setup on 8. > > I've attached the log_summary for 1 processor and 8 processors. You > can also see that the solve time is completely eclipsed by the setup > time. > > Regards, > John > From john.fettig at gmail.com Fri Aug 17 09:59:32 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 10:59:32 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> Message-ID: Mark, Thanks for your response, comments inline: On Fri, Aug 17, 2012 at 10:50 AM, Mark F. Adams wrote: > Yes John, the GAMG setup is not great. So we know there are problems. We do have a new staff person that will be working on matrix product methods soon. Matrix products are hard to make fast and GAMG setup relies on them quite a bit. > > A few things to note: > > 1) This is in the mesh setup phase, so many applications can amortize this cost. (MatPtAP is a "matrix" setup cost and so it is amortized for linear problems or non-full Newton solves, but not for full nonlinear solves). I'd be willing to live with the MatPtAP time, it is the MatTransposeMatMult time that is really poor. Any idea why the latter is so slow? > 3) This test is getting less than 2x speedup in KSPSolve for 8x processors. So this problem looks hard: small or poorly partitioned, and not in the range of where we want people to run to get good performance. It's about 725k unknowns, and it was run with ex10 from the ksp tutorials (i.e. loaded from a file and solved). It is possible the partitioning isn't great. > 4) I have found that the setup times are about twice that of ML, which uses a similar algorithm, and about 5x slower than hypre, which uses a very different algorithm. So if you can not amortize theses setup costs then ML or hypre would probably work better for you. I think you actually have this backwards, or your experience is different from mine. In my experience ML's setup is much faster than that of hypre. John From jedbrown at mcs.anl.gov Fri Aug 17 10:08:34 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 10:08:34 -0500 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> Message-ID: On Fri, Aug 17, 2012 at 9:59 AM, John Fettig wrote: > I'd be willing to live with the MatPtAP time, it is the > MatTransposeMatMult time that is really poor. Any idea why the latter > is so slow? > Mark, for a symmetric graph, how about using MatMatMult() here? Even for a non-symmetric problem, MatMatTransposeMult() is faster than MatTransposeMatMult(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Fri Aug 17 10:12:43 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 17 Aug 2012 08:12:43 -0700 Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> Message-ID: <502E5F6B.1050206@uci.edu> Thank you. Does this mean that Fortran compiler invoked by the PETSc Makefile will compile to code with the 64 bit integer option so that also Fortran Integer will be 64 bit? Michele On 08/17/2012 07:43 AM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 9:40 AM, Michele Rosso > wrote: > > I want to build my application with 64 bit integer support > because of the size of the problem I am working on. > I compiled petsc with the option /--with-64-bit-indices=1/. > In my Fortran code some petsc vectors are initialize by taking > values from Fortran arrays. > I used PetscInt indexes to reference both petsc and fortran vectors. > It used to work with 32 bit indexes. > Will it still be the case for the 64 bit case? > > > As long as you use PetscInt for every integer that you pass to and > from PETSc, it will work automatically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 10:16:47 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 10:16:47 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E5F6B.1050206@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: > Does this mean that Fortran compiler invoked by the PETSc Makefile will > compile to code with the 64 bit integer option so that also Fortran > Integer will be 64 bit? NO, that would be bad because it breaks library interfaces. You have to use the right types internally. If you use PetscInt everywhere, then you'll be fine. If you mix types, you have to find out where to convert. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 17 10:16:54 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 10:16:54 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E5F6B.1050206@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: > Thank you. > Does this mean that Fortran compiler invoked by the PETSc Makefile will > compile to code with the 64 bit integer option so that also Fortran > Integer will be 64 bit? > Yes Matt > Michele > > On 08/17/2012 07:43 AM, Jed Brown wrote: > > On Fri, Aug 17, 2012 at 9:40 AM, Michele Rosso wrote: > >> I want to build my application with 64 bit integer support >> because of the size of the problem I am working on. >> I compiled petsc with the option *--with-64-bit-indices=1*. >> In my Fortran code some petsc vectors are initialize by taking >> values from Fortran arrays. >> I used PetscInt indexes to reference both petsc and fortran vectors. >> It used to work with 32 bit indexes. >> Will it still be the case for the 64 bit case? >> > > As long as you use PetscInt for every integer that you pass to and from > PETSc, it will work automatically. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Aug 17 10:19:41 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 17 Aug 2012 11:19:41 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> Message-ID: <22A479D9-6DFA-411D-BE90-1990995B9E3C@columbia.edu> On Aug 17, 2012, at 10:59 AM, John Fettig wrote: > Mark, > > Thanks for your response, comments inline: > > On Fri, Aug 17, 2012 at 10:50 AM, Mark F. Adams wrote: >> Yes John, the GAMG setup is not great. So we know there are problems. We do have a new staff person that will be working on matrix product methods soon. Matrix products are hard to make fast and GAMG setup relies on them quite a bit. >> >> A few things to note: >> >> 1) This is in the mesh setup phase, so many applications can amortize this cost. (MatPtAP is a "matrix" setup cost and so it is amortized for linear problems or non-full Newton solves, but not for full nonlinear solves). > > I'd be willing to live with the MatPtAP time, it is the > MatTransposeMatMult time that is really poor. Any idea why the latter > is so slow? Its a complex problem and I did not work on the code. > >> 3) This test is getting less than 2x speedup in KSPSolve for 8x processors. So this problem looks hard: small or poorly partitioned, and not in the range of where we want people to run to get good performance. > > It's about 725k unknowns, and it was run with ex10 from the ksp > tutorials (i.e. loaded from a file and solved). It is possible the > partitioning isn't great. This big enough and the partitioning is fine for our examples. If this is a shared memory machine then you are probably seeing memory bandwidth effects (one processor gets the memory bandwidth of the whole machine and rocks). If this is distributed memory then it looks like you have a crapy network. > >> 4) I have found that the setup times are about twice that of ML, which uses a similar algorithm, and about 5x slower than hypre, which uses a very different algorithm. So if you can not amortize theses setup costs then ML or hypre would probably work better for you. > > I think you actually have this backwards, or your experience is > different from mine. In my experience ML's setup is much faster than > that of hypre. 2D and 3D can make a big difference. Hypre has problems in 3D (that can be addressed to some degree with parameters). But yes my numbers were from limited experiments a while ago so you probably have a better idea than me. Mark > > John > From balay at mcs.anl.gov Fri Aug 17 10:21:26 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 17 Aug 2012 10:21:26 -0500 (CDT) Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> Message-ID: On Fri, 17 Aug 2012, Jed Brown wrote: > On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: > > > Does this mean that Fortran compiler invoked by the PETSc Makefile will > > compile to code with the 64 bit integer option so that also Fortran > > Integer will be 64 bit? > > > NO, that would be bad because it breaks library interfaces. > > You have to use the right types internally. If you use PetscInt everywhere, > then you'll be fine. If you mix types, you have to find out where to > convert. A couple of notes. - PETSc does not use -i8 compiler flag. It uses '#define PetscInt integer*8'. - Some fortran compilers default to 32bit indexing for arrays [g95?] but I think most compilers do use 64bit array indexing. - With /--with-64-bit-indices=1/ one can have the global problem dimentions exceed 32bit indices - but it should still be within 32bit limit for the local parts. [as we still use 32bit blas calls]. Satish From mark.adams at columbia.edu Fri Aug 17 10:27:43 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 17 Aug 2012 11:27:43 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> Message-ID: <5C15C499-71D8-4F78-8DFB-ADEB29ED85E8@columbia.edu> On Aug 17, 2012, at 11:08 AM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 9:59 AM, John Fettig wrote: > I'd be willing to live with the MatPtAP time, it is the > MatTransposeMatMult time that is really poor. Any idea why the latter > is so slow? > > Mark, for a symmetric graph, how about using MatMatMult() here? Even for a non-symmetric problem, MatMatTransposeMult() is faster than MatTransposeMatMult(). The code has: if( pc_gamg_agg->square_graph ) { /* ierr = MatMatTransposeMult( Gmat1, Gmat1, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &Gmat2 ); */ ierr = MatTransposeMatMult( Gmat1, Gmat1, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &Gmat2 ); CHKERRQ(ierr); } I thought MatMatTransposeMult would be faster also, but my experiments were inconclusive ... and I figure this has got to change so I did not worry too much about optimizing it. I don't know if a matrix is symmetric and I've not wanted to rely on the user telling me, but I'm willing to add this logic if it is useful. John: could you perhaps test MatMatMult and MatMatTransposeMult so we could get some concrete numbers? Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vincent.De-Groof at uibk.ac.at Fri Aug 17 10:25:47 2012 From: Vincent.De-Groof at uibk.ac.at (De Groof, Vincent) Date: Fri, 17 Aug 2012 15:25:47 +0000 Subject: [petsc-users] Segmentation fault when using matlab interface In-Reply-To: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> Message-ID: <17A78B9D13564547AC894B88C1596747035169@XMBX4.uibk.ac.at> Well, basically the simplest program I have the problem with is initializing and finalizing petsc. % Set environment variables. setenv('PETSC_DIR', '/usr/local/petsc-3.3-p2/') setenv('PETSC_ARCH','arch-linux2-c-debug') % Call PETSc. PetscInitialize(); PetscFinalize(); thanks. ________________________________ Von: De Groof, Vincent Gesendet: Freitag, 17. August 2012 10:25 An: petsc-users at mcs.anl.gov Betreff: Segmentation fault when using matlab interface Hello, I'm trying to use the Petsc objects directly from Matlab. However, when testing the examples in the /bin/matlab/classes/examples/tutorials-directory, I get a segmentation fault (core dump) error. Unfortunately, I don't get more information then this. After running some different situations, I know the following: I ran matlab both with and without desktop. The only error message I get is the segmentation fault. Sometimes it doesn't give the segmentation error after running the example. If so, it will cause a segmentation fault upon entering the next command in the matlab shell. If I quit matlab immediately after running an example without segmentation fault, my unix shell will hang. I know it's kind of vague, but this is all I know now. Any suggestions to look at to solve this problem? thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Aug 17 10:43:02 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 11:43:02 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: <5C15C499-71D8-4F78-8DFB-ADEB29ED85E8@columbia.edu> References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> <5C15C499-71D8-4F78-8DFB-ADEB29ED85E8@columbia.edu> Message-ID: On Fri, Aug 17, 2012 at 11:27 AM, Mark F. Adams wrote: > John: could you perhaps test MatMatMult and MatMatTransposeMult so we could > get some concrete numbers? For MatMatTransposeMult I get this error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: MatMatTransposeMult not supported for A of type mpiaij! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 -0700 For MatMatMult, my problem is not symmetric. It doesn't look like MatTranspose() followed by MatMatMult is a viable option. John From benjamin.kirk-1 at nasa.gov Fri Aug 17 10:47:02 2012 From: benjamin.kirk-1 at nasa.gov (Kirk, Benjamin (JSC-EG311)) Date: Fri, 17 Aug 2012 10:47:02 -0500 Subject: [petsc-users] Updating ghost mapping for an existing vector? Message-ID: Hello - I've got a contact problem with moving unstructured meshes. The issue here is that the required ghost indices can change during the course of a simulation. Is there a mechanism to reset the ghost indices for an existing Vec object? Thanks, -Ben From jedbrown at mcs.anl.gov Fri Aug 17 10:49:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 10:49:43 -0500 Subject: [petsc-users] Updating ghost mapping for an existing vector? In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 10:47 AM, Kirk, Benjamin (JSC-EG311) < benjamin.kirk-1 at nasa.gov> wrote: > Hello - > > I've got a contact problem with moving unstructured meshes. The issue here > is that the required ghost indices can change during the course of a > simulation. Is there a mechanism to reset the ghost indices for an > existing > Vec object? > It can't be done at the system level without reallocation (assuming the number of ghost points has changed). This is a weakness of using ghost vectors compared to explicit local work vectors. How often do you want to do this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Aug 17 10:50:14 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 17 Aug 2012 10:50:14 -0500 (CDT) Subject: [petsc-users] Segmentation fault when using matlab interface In-Reply-To: <17A78B9D13564547AC894B88C1596747035169@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> <17A78B9D13564547AC894B88C1596747035169@XMBX4.uibk.ac.at> Message-ID: one issue with linking to 64bit matlab is: it internally links in with 64int mlk/blas [packaged in matlab]. So I think we have to use this mkl with petsc [along with -known-64-bit-blas-indices=1 option] But there should be no calls to blas with just petscinitialize/finalize. Perhaps its possible to run this in a debugger to determine the exact reason for SEGV? Satish On Fri, 17 Aug 2012, De Groof, Vincent wrote: > Well, basically the simplest program I have the problem with is initializing and finalizing petsc. > > % Set environment variables. > setenv('PETSC_DIR', '/usr/local/petsc-3.3-p2/') > setenv('PETSC_ARCH','arch-linux2-c-debug') > > % Call PETSc. > PetscInitialize(); > PetscFinalize(); > > > thanks. > ________________________________ > Von: De Groof, Vincent > Gesendet: Freitag, 17. August 2012 10:25 > An: petsc-users at mcs.anl.gov > Betreff: Segmentation fault when using matlab interface > > Hello, > > I'm trying to use the Petsc objects directly from Matlab. However, when testing the examples in the /bin/matlab/classes/examples/tutorials-directory, I get a segmentation fault (core dump) error. Unfortunately, I don't get more information then this. After running some different situations, I know the following: > I ran matlab both with and without desktop. The only error message I get is the segmentation fault. Sometimes it doesn't give the segmentation error after running the example. If so, it will cause a segmentation fault upon entering the next command in the matlab shell. If I quit matlab immediately after running an example without segmentation fault, my unix shell will hang. > > > I know it's kind of vague, but this is all I know now. Any suggestions to look at to solve this problem? > > > thanks. > From mark.adams at columbia.edu Fri Aug 17 10:53:58 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 17 Aug 2012 11:53:58 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> <5C15C499-71D8-4F78-8DFB-ADEB29ED85E8@columbia.edu> Message-ID: <57CC222B-21FB-41D4-A95C-4C808B19BF5D@columbia.edu> Thanks for trying John, I had forgotten about MatMatTransposeMult not working in parallel. And yes it is fair to say that MatTranspose() followed by MatMatMult would not be faster. Mark On Aug 17, 2012, at 11:43 AM, John Fettig wrote: > On Fri, Aug 17, 2012 at 11:27 AM, Mark F. Adams wrote: >> John: could you perhaps test MatMatMult and MatMatTransposeMult so we could >> get some concrete numbers? > > For MatMatTransposeMult I get this error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: MatMatTransposeMult not supported for A of type mpiaij! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 > 2012 -0700 > > For MatMatMult, my problem is not symmetric. It doesn't look like > MatTranspose() followed by MatMatMult is a viable option. > > John > From john.fettig at gmail.com Fri Aug 17 10:52:35 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 11:52:35 -0400 Subject: [petsc-users] GAMG setup scalability In-Reply-To: References: <3A8B9026-0686-448E-89FF-41E4C39D3783@columbia.edu> <5C15C499-71D8-4F78-8DFB-ADEB29ED85E8@columbia.edu> Message-ID: For what it's worth, I've attached the results from np=1 for MatMatTransposeMult. It is significantly slower, although it converges in 2 fewer iterations with a smaller residual. John On Fri, Aug 17, 2012 at 11:43 AM, John Fettig wrote: > On Fri, Aug 17, 2012 at 11:27 AM, Mark F. Adams wrote: >> John: could you perhaps test MatMatMult and MatMatTransposeMult so we could >> get some concrete numbers? > > For MatMatTransposeMult I get this error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: MatMatTransposeMult not supported for A of type mpiaij! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 > 2012 -0700 > > For MatMatMult, my problem is not symmetric. It doesn't look like > MatTranspose() followed by MatMatMult is a viable option. > > John -------------- next part -------------- A non-text attachment was scrubbed... Name: mmtm_np1.out Type: application/octet-stream Size: 14909 bytes Desc: not available URL: From knepley at gmail.com Fri Aug 17 10:55:43 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 10:55:43 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: > >> Does this mean that Fortran compiler invoked by the PETSc Makefile will >> compile to code with the 64 bit integer option so that also Fortran >> Integer will be 64 bit? > > > NO, that would be bad because it breaks library interfaces. > > You have to use the right types internally. If you use PetscInt > everywhere, then you'll be fine. If you mix types, you have to find out > where to convert. > I meant that if you use PetscInt in Fortran it was also be a 64-bit integer. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Fri Aug 17 11:04:07 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 17 Aug 2012 09:04:07 -0700 Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> Message-ID: <502E6B77.2080809@uci.edu> Thank you. How can I convert? Michele On 08/17/2012 08:55 AM, Matthew Knepley wrote: > On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown > wrote: > > On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso > wrote: > > Does this mean that Fortran compiler invoked by the PETSc > Makefile will > compile to code with the 64 bit integer option so that also > Fortran Integer will be 64 bit? > > > NO, that would be bad because it breaks library interfaces. > > You have to use the right types internally. If you use PetscInt > everywhere, then you'll be fine. If you mix types, you have to > find out where to convert. > > > I meant that if you use PetscInt in Fortran it was also be a 64-bit > integer. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 17 11:08:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 11:08:01 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E6B77.2080809@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 11:04 AM, Michele Rosso wrote: > Thank you. > How can I convert? > Don't use integer, use PetscInt. See any PETSc example. Matt > Michele > > On 08/17/2012 08:55 AM, Matthew Knepley wrote: > > On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown wrote: > >> On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: >> >>> Does this mean that Fortran compiler invoked by the PETSc Makefile will >>> compile to code with the 64 bit integer option so that also Fortran >>> Integer will be 64 bit? >> >> >> NO, that would be bad because it breaks library interfaces. >> >> You have to use the right types internally. If you use PetscInt >> everywhere, then you'll be fine. If you mix types, you have to find out >> where to convert. >> > > I meant that if you use PetscInt in Fortran it was also be a 64-bit > integer. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Aug 17 11:11:44 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 17 Aug 2012 11:11:44 -0500 (CDT) Subject: [petsc-users] 64 bit integers In-Reply-To: <502E6B77.2080809@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> Message-ID: If you are calling a petsc function that expects PetscInt - then the following coulde would be a bug: /* VecSetSizes(Vec,PetscInt,PetscInt) */ integer nlocal,nglobal Vec v call VecSetSizes(v,nlocal,nglobal) It should be: PetscInt nlocal,nglobal Vec v call VecSetSizes(v,nlocal,nglobal) Satish On Fri, 17 Aug 2012, Michele Rosso wrote: > Thank you. > How can I convert? > > Michele > > On 08/17/2012 08:55 AM, Matthew Knepley wrote: > > On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown > > wrote: > > > > On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso > > wrote: > > > > Does this mean that Fortran compiler invoked by the PETSc > > Makefile will > > compile to code with the 64 bit integer option so that also > > Fortran Integer will be 64 bit? > > > > > > NO, that would be bad because it breaks library interfaces. > > > > You have to use the right types internally. If you use PetscInt > > everywhere, then you'll be fine. If you mix types, you have to > > find out where to convert. > > > > > > I meant that if you use PetscInt in Fortran it was also be a 64-bit integer. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their experiments > > is infinitely more interesting than any results to which their experiments > > lead. > > -- Norbert Wiener > > From mrosso at uci.edu Fri Aug 17 11:12:31 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 17 Aug 2012 09:12:31 -0700 Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> Message-ID: <502E6D6F.2060703@uci.edu> The problem is that I use petsc only in a small part of my code. Given that I have integers everywhere and I use other libraries as well, I cannot/ don't want to re-define all the integers I have as PetscInt. Michele On 08/17/2012 09:08 AM, Matthew Knepley wrote: > On Fri, Aug 17, 2012 at 11:04 AM, Michele Rosso > wrote: > > Thank you. > How can I convert? > > > Don't use integer, use PetscInt. See any PETSc example. > > Matt > > Michele > > On 08/17/2012 08:55 AM, Matthew Knepley wrote: >> On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown > > wrote: >> >> On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso >> > wrote: >> >> Does this mean that Fortran compiler invoked by the PETSc >> Makefile will >> compile to code with the 64 bit integer option so that >> also Fortran Integer will be 64 bit? >> >> >> NO, that would be bad because it breaks library interfaces. >> >> You have to use the right types internally. If you use >> PetscInt everywhere, then you'll be fine. If you mix types, >> you have to find out where to convert. >> >> >> I meant that if you use PetscInt in Fortran it was also be a >> 64-bit integer. >> >> Matt >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 17 11:24:45 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 11:24:45 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E6D6F.2060703@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> <502E6D6F.2060703@uci.edu> Message-ID: On Fri, Aug 17, 2012 at 11:12 AM, Michele Rosso wrote: > The problem is that I use petsc only in a small part of my code. > Given that I have integers everywhere and I use other libraries as well, > I cannot/ don't want to re-define all the integers I have as PetscInt. > -with-64-bit-integers is a feature of PETSc, not random code. We cannot control what is done in other libraries. We do our best by giving you types that do this. Matt > Michele > > > On 08/17/2012 09:08 AM, Matthew Knepley wrote: > > On Fri, Aug 17, 2012 at 11:04 AM, Michele Rosso wrote: > >> Thank you. >> How can I convert? >> > > Don't use integer, use PetscInt. See any PETSc example. > > Matt > > >> Michele >> >> On 08/17/2012 08:55 AM, Matthew Knepley wrote: >> >> On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown wrote: >> >>> On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: >>> >>>> Does this mean that Fortran compiler invoked by the PETSc Makefile will >>>> compile to code with the 64 bit integer option so that also Fortran >>>> Integer will be 64 bit? >>> >>> >>> NO, that would be bad because it breaks library interfaces. >>> >>> You have to use the right types internally. If you use PetscInt >>> everywhere, then you'll be fine. If you mix types, you have to find out >>> where to convert. >>> >> >> I meant that if you use PetscInt in Fortran it was also be a 64-bit >> integer. >> >> Matt >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Fri Aug 17 11:32:02 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 17 Aug 2012 09:32:02 -0700 Subject: [petsc-users] 64 bit integers In-Reply-To: References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> <502E6D6F.2060703@uci.edu> Message-ID: <502E7202.2040502@uci.edu> Thanks a lot for your help. Michele On 08/17/2012 09:24 AM, Matthew Knepley wrote: > On Fri, Aug 17, 2012 at 11:12 AM, Michele Rosso > wrote: > > The problem is that I use petsc only in a small part of my code. > Given that I have integers everywhere and I use other libraries as > well, > I cannot/ don't want to re-define all the integers I have as > PetscInt. > > > -with-64-bit-integers is a feature of PETSc, not random code. We cannot > control what is done in other libraries. We do our best by giving you > types > that do this. > > Matt > > Michele > > > On 08/17/2012 09:08 AM, Matthew Knepley wrote: >> On Fri, Aug 17, 2012 at 11:04 AM, Michele Rosso > > wrote: >> >> Thank you. >> How can I convert? >> >> >> Don't use integer, use PetscInt. See any PETSc example. >> >> Matt >> >> Michele >> >> On 08/17/2012 08:55 AM, Matthew Knepley wrote: >>> On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown >>> > wrote: >>> >>> On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso >>> > wrote: >>> >>> Does this mean that Fortran compiler invoked by the >>> PETSc Makefile will >>> compile to code with the 64 bit integer option so >>> that also Fortran Integer will be 64 bit? >>> >>> >>> NO, that would be bad because it breaks library interfaces. >>> >>> You have to use the right types internally. If you use >>> PetscInt everywhere, then you'll be fine. If you mix >>> types, you have to find out where to convert. >>> >>> >>> I meant that if you use PetscInt in Fortran it was also be a >>> 64-bit integer. >>> >>> Matt >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 17 11:33:51 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 11:33:51 -0500 Subject: [petsc-users] 64 bit integers In-Reply-To: <502E6D6F.2060703@uci.edu> References: <502E57F7.8020601@uci.edu> <502E5F6B.1050206@uci.edu> <502E6B77.2080809@uci.edu> <502E6D6F.2060703@uci.edu> Message-ID: On Aug 17, 2012, at 11:12 AM, Michele Rosso wrote: > The problem is that I use petsc only in a small part of my code. > Given that I have integers everywhere and I use other libraries as well, > I cannot/ don't want to re-define all the integers I have as PetscInt. > > Michele So do you want all the rest of the integers in your code to be 64 bit integers or can they still be 32 bit integers? If so you do things like integer myoldint PetscInt mypetscversionofoldint mypetscversionofoldint = myoldit call PETSc function with mypetscversionofallint variables. Note, course, that myoldint limits the size integers you can pass to PETSc Or do you want to automagically make all integer in your FORTRAN code suddenly be 64 bit int?? If so you can use the crazy FORTRAN command line flags that cause the Fortran compiler to make everything declared integer to be 64 bit. Now all your integers are 64 bit so you can pass them directly to PETSc functions. We don't generally recommend this approach because it is fraught with danger and the Fortran compiler sometimes does other stuff like promoting the length of other things at the same time. If you go this route you need to manually stick that special fortran flag into your makefiles in the FFLAGS variable. Good luck. Barry > > > On 08/17/2012 09:08 AM, Matthew Knepley wrote: >> On Fri, Aug 17, 2012 at 11:04 AM, Michele Rosso wrote: >> Thank you. >> How can I convert? >> >> Don't use integer, use PetscInt. See any PETSc example. >> >> Matt >> >> Michele >> >> On 08/17/2012 08:55 AM, Matthew Knepley wrote: >>> On Fri, Aug 17, 2012 at 10:16 AM, Jed Brown wrote: >>> On Fri, Aug 17, 2012 at 10:12 AM, Michele Rosso wrote: >>> Does this mean that Fortran compiler invoked by the PETSc Makefile will >>> compile to code with the 64 bit integer option so that also Fortran Integer will be 64 bit? >>> >>> NO, that would be bad because it breaks library interfaces. >>> >>> You have to use the right types internally. If you use PetscInt everywhere, then you'll be fine. If you mix types, you have to find out where to convert. >>> >>> I meant that if you use PetscInt in Fortran it was also be a 64-bit integer. >>> >>> Matt >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From pwu at mymail.mines.edu Fri Aug 17 12:13:15 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Fri, 17 Aug 2012 11:13:15 -0600 Subject: [petsc-users] VecGetValues get vector values at arbitrary locations Message-ID: Dear PetSc developers and users, I want to access a Petsc Vec v at index n whose value is not necessarily stored in the same processor. I noted that the current VecGetValues cannot do that. Do you have any suggestions on how I can access v(n) from arbitrary processors(in the same communicator of course)? Thanks, best wishes, Panruo Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Fri Aug 17 12:16:44 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Fri, 17 Aug 2012 19:16:44 +0200 Subject: [petsc-users] VecGetValues get vector values at arbitrary locations In-Reply-To: References: Message-ID: Dear Wu, This might help: http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-to-seq-vec How do I collect all the values from a parallel PETSc vector into a sequential vector on each processor? How do I collect all the values from a parallel PETSc vector into a vector on the zeroth processor? Regards, Zhengyong On Fri, Aug 17, 2012 at 7:13 PM, Panruo Wu wrote: > Dear PetSc developers and users, > > I want to access a Petsc Vec v at index n whose value > is not necessarily stored in the same processor. > I noted that the current VecGetValues cannot do that. Do you have > any suggestions on how I can access v(n) from arbitrary > processors(in the same communicator of course)? > > Thanks, > > best wishes, > Panruo Wu > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Fri Aug 17 12:32:43 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Fri, 17 Aug 2012 11:32:43 -0600 Subject: [petsc-users] VecGetValues get vector values at arbitrary locations In-Reply-To: References: Message-ID: Thank you Zhengyong! That's very useful. Panruo On Fri, Aug 17, 2012 at 11:16 AM, RenZhengYong wrote: > Dear Wu, > > This might help: > > http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-to-seq-vec > How do I collect all the values from a parallel PETSc vector into a > sequential vector on each processor? > How do I collect all the values from a parallel PETSc vector into a vector > on the zeroth processor? > Regards, > Zhengyong > > > On Fri, Aug 17, 2012 at 7:13 PM, Panruo Wu wrote: > >> Dear PetSc developers and users, >> >> I want to access a Petsc Vec v at index n whose value >> is not necessarily stored in the same processor. >> I noted that the current VecGetValues cannot do that. Do you have >> any suggestions on how I can access v(n) from arbitrary >> processors(in the same communicator of course)? >> >> Thanks, >> >> best wishes, >> Panruo Wu >> > > > > -- > Zhengyong Ren > AUG Group, Institute of Geophysics > Department of Geosciences, ETH Zurich > NO H 47 Sonneggstrasse 5 > CH-8092, Z?rich, Switzerland > Tel: +41 44 633 37561 > e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > Gmail: renzhengyong at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Aug 17 13:01:40 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 14:01:40 -0400 Subject: [petsc-users] Scatter context for MatMult Message-ID: Is there a scatter context available someplace that would allow me to gather all of the elements of a vector that correspond to the ones needed for a MatMult (MPIAIJ matrices)? When you do a MatMult, I presume it first scatters/gathers the ghost elements of the vector being multiplied, can I use this scatter for a vectors outside of the MatMult? Thanks, John From john.fettig at gmail.com Fri Aug 17 13:05:11 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 14:05:11 -0400 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: Is this the role of MatGetGhosts? John On Fri, Aug 17, 2012 at 2:01 PM, John Fettig wrote: > Is there a scatter context available someplace that would allow me to > gather all of the elements of a vector that correspond to the ones > needed for a MatMult (MPIAIJ matrices)? When you do a MatMult, I > presume it first scatters/gathers the ghost elements of the vector > being multiplied, can I use this scatter for a vectors outside of the > MatMult? > > Thanks, > John From alexvg77 at gmail.com Fri Aug 17 13:12:33 2012 From: alexvg77 at gmail.com (Alexander Goncharov) Date: Fri, 17 Aug 2012 11:12:33 -0700 Subject: [petsc-users] Mixed precision Message-ID: <1345227153.20884.23.camel@DeathStar> Hi all! Is it possible in PETSC to use double and single precision matrices in the same code? Thank you, Alex. From benjamin.kirk-1 at nasa.gov Fri Aug 17 13:28:20 2012 From: benjamin.kirk-1 at nasa.gov (Kirk, Benjamin (JSC-EG311)) Date: Fri, 17 Aug 2012 13:28:20 -0500 Subject: [petsc-users] Updating ghost mapping for an existing vector? In-Reply-To: Message-ID: >> I've got a contact problem with moving unstructured meshes. ?The issue here >> is that the required ghost indices can change during the course of a >> simulation. ?Is there a mechanism to reset the ghost indices for an existing >> Vec object? > > It can't be done at the system level without reallocation (assuming the number > of ghost points has changed). This is a weakness of using ghost vectors > compared to explicit local work vectors. How often do you want to do this? Thanks for the reply. Probably once per time step. I'll work around it then by writing an in-place vector replacement function that takes the new ghost points and does the required copy... -Ben From jedbrown at mcs.anl.gov Fri Aug 17 13:28:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 13:28:41 -0500 Subject: [petsc-users] VecGetValues get vector values at arbitrary locations In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 12:13 PM, Panruo Wu wrote: > Dear PetSc developers and users, > > I want to access a Petsc Vec v at index n whose value > is not necessarily stored in the same processor. > I noted that the current VecGetValues cannot do that. Do you have > any suggestions on how I can access v(n) from arbitrary > processors(in the same communicator of course)? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreate.html See also the user's manual chapter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 13:33:55 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 13:33:55 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 1:05 PM, John Fettig wrote: > Is this the role of MatGetGhosts? > That's the intent, but it looks like this function was only implemented for MPIBAIJ. The actual implementation of MatMult_MPI*AIJ is to start the scatter, apply the diagonal block, finish the scatter, and apply the off-diagonal part. Why do you want raw access? If you really need, you can either use MatMPIAIJGetSeqAIJ() or just include the private header. > > John > > On Fri, Aug 17, 2012 at 2:01 PM, John Fettig > wrote: > > Is there a scatter context available someplace that would allow me to > > gather all of the elements of a vector that correspond to the ones > > needed for a MatMult (MPIAIJ matrices)? When you do a MatMult, I > > presume it first scatters/gathers the ghost elements of the vector > > being multiplied, can I use this scatter for a vectors outside of the > > MatMult? > > > > Thanks, > > John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 13:34:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 13:34:57 -0500 Subject: [petsc-users] Mixed precision In-Reply-To: <1345227153.20884.23.camel@DeathStar> References: <1345227153.20884.23.camel@DeathStar> Message-ID: On Fri, Aug 17, 2012 at 1:12 PM, Alexander Goncharov wrote: > Hi all! > > Is it possible in PETSC to use double and single precision matrices in > the same code? > Not unless you define your own reduced-precision MATSHELL. > > Thank you, > Alex. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 13:37:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 13:37:13 -0500 Subject: [petsc-users] Updating ghost mapping for an existing vector? In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 1:28 PM, Kirk, Benjamin (JSC-EG311) < benjamin.kirk-1 at nasa.gov> wrote: > Thanks for the reply. Probably once per time step. I'll work around it > then by writing an in-place vector replacement function that takes the new > ghost points and does the required copy... > When I visited Idaho last week, we discussed an interface for dynamic AMR using PETSc TS. I plan to implement that sometime this fall. In any case, if you have to reallocate the matrix due to different sparsity pattern, it's not a big deal to just SNESReset() and replace the vectors as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Aug 17 13:48:12 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 14:48:12 -0400 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: > Why do you want raw access? I'm trying to write my own RAP operator for AMG. > If you really need, you can either use MatMPIAIJGetSeqAIJ() or just include > the private header. I presume you mean MatMPIAIJGetLocalMat. This isn't quite what I want. I don't want the local matrix, I want to scatter a vector according to the ghost values defined by the sparsity pattern of the parallel matrix. Could you give me a little more help with using the latter? I see VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); (*a->A->ops->mult)(a->A,xx,yy); VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); (*a->B->ops->multadd)(a->B,a->lvec,yy,yy); So I guess a->Mvctx is what I want. How do I figure out the mapping of indices in a->lvec? John From alexvg77 at gmail.com Fri Aug 17 13:58:26 2012 From: alexvg77 at gmail.com (Alexander Goncharov) Date: Fri, 17 Aug 2012 11:58:26 -0700 Subject: [petsc-users] Mixed precision In-Reply-To: References: <1345227153.20884.23.camel@DeathStar> Message-ID: <1345229906.20884.24.camel@DeathStar> Thank you, Jed! On Fri, 2012-08-17 at 13:34 -0500, Jed Brown wrote: > On Fri, Aug 17, 2012 at 1:12 PM, Alexander Goncharov > wrote: > Hi all! > > Is it possible in PETSC to use double and single precision > matrices in > the same code? > > > Not unless you define your own reduced-precision MATSHELL. > > > Thank you, > Alex. > > From jedbrown at mcs.anl.gov Fri Aug 17 13:59:51 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 13:59:51 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 1:48 PM, John Fettig wrote: > On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: > > Why do you want raw access? > > I'm trying to write my own RAP operator for AMG. > > > If you really need, you can either use MatMPIAIJGetSeqAIJ() or just > include > > the private header. > > I presume you mean MatMPIAIJGetLocalMat. This isn't quite what I > want. I don't want the local matrix, I want to scatter a vector > according to the ghost values defined by the sparsity pattern of the > parallel matrix. > PetscErrorCode MatMPIAIJGetSeqAIJ(Mat A,Mat *Ad,Mat *Ao,PetscInt **colmap); But if you are writing an RAP, you should just include src/mat/impls/aij/mpi/mpiaij.h. You can set your implementation on an existing matrix using MatShellSetOperation(A, MATOP_PTAP, (void(*)(void))YourPtAPFunction). > Could you give me a little more help with using the latter? I see > > VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); > (*a->A->ops->mult)(a->A,xx,yy); > VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); > (*a->B->ops->multadd)(a->B,a->lvec,yy,yy); > > So I guess a->Mvctx is what I want. How do I figure out the mapping > of indices in a->lvec? > a->garray. MatPtAP is not an easy operation to write. You'll have to get comfortable with diving into the source to understand what's happening. > > John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Fri Aug 17 14:06:01 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Fri, 17 Aug 2012 21:06:01 +0200 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: References: <502DFC8B.4080800@tu-dresden.de> Message-ID: <502E9619.3080807@tu-dresden.de> Am 17.08.2012 16:24, schrieb Matthew Knepley: > On Fri, Aug 17, 2012 at 3:10 AM, Thomas Witkowski > > wrote: > > I want to solve some (weakly) coupled system of equations of the > following form: > > A B u > . = ..... > 0 C v > > > so, C is the discrete Laplacian and A and B are some more > complicated operators (I make use of linear finite elements). All > boundary conditions are periodic, so the unknown v is determined > only up to a constant. A and B contain both the identity operator, > so u is fixed. Now I want to solve the system on the whole (there > are reasons to do it in this way!) and I must provide information > about the nullspace to the solver. When I am right, to provide the > correct nullspace I must solve one equation with A. Is there any > way in PETSc to circumvent the problem? > > > If I understand you correctly, your null space vector is (0 I). I use > the same null space for SNES ex62. (0 I) cannot be an element of the null space, as multiplying it with the matrix results in a non-zero vector. Or am I totally wrong about null spaces of matrices? Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 17 14:11:04 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 17 Aug 2012 12:11:04 -0700 Subject: [petsc-users] issue with different versions of petsc-dev Message-ID: Hi guys, I just realize my code breaks with new versions of petsc-dev. I use MatCreateMPIAIJWithSplitArrays() funtions and it works with this verison: Petsc Development HG revision: b27fc9457bdc002c5a5de2b07797ae1a93094370 HG Date: Thu May 03 13:42:15 2012 -0500 but breaks with: Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 -0700 Here's the error code: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Column entry number 1 (actual colum 2) in row 3 is not sorted! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 -0700 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./FVM on a arch-linu named mohammad-desktop by mohammad Fri Aug 17 12:06:15 2012 [0]PETSC ERROR: Libraries linked from /home/mohammad/soft/petsc-dev_new/petsc-dev/arch-linux2-cxx-debug/lib [0]PETSC ERROR: Configure run at Fri Aug 17 11:45:31 2012 [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-mpi-dir=/usr/bin --download-f-blas-lapack=1 --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4268 in /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatCreateMPIAIJWithSplitArrays() line 5731 in /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: setLinearSystemMatrix() line 84 in "unknowndirectory/"../../../CASL/lib/algebra/petscLinearSolver.cpp Is there anything changed regarding numbering of diagonal and off-diagonal entries when using the said function between these two versions? Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 17 14:11:31 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 14:11:31 -0500 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: <502E9619.3080807@tu-dresden.de> References: <502DFC8B.4080800@tu-dresden.de> <502E9619.3080807@tu-dresden.de> Message-ID: On Fri, Aug 17, 2012 at 2:06 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Am 17.08.2012 16:24, schrieb Matthew Knepley: > > On Fri, Aug 17, 2012 at 3:10 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> I want to solve some (weakly) coupled system of equations of the >> following form: >> >> A B u >> . = ..... >> 0 C v >> >> >> so, C is the discrete Laplacian and A and B are some more complicated >> operators (I make use of linear finite elements). All boundary conditions >> are periodic, so the unknown v is determined only up to a constant. A and B >> contain both the identity operator, so u is fixed. Now I want to solve the >> system on the whole (there are reasons to do it in this way!) and I must >> provide information about the nullspace to the solver. When I am right, to >> provide the correct nullspace I must solve one equation with A. Is there >> any way in PETSc to circumvent the problem? > > > If I understand you correctly, your null space vector is (0 I). I use > the same null space for SNES ex62. > > (0 I) cannot be an element of the null space, as multiplying it with the > matrix results in a non-zero vector. Or am I totally wrong about null > spaces of matrices? > Maybe you could as your question again. I am not understanding what you want. Matt > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 14:18:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 14:18:11 -0500 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 2:11 PM, Mohammad Mirzadeh wrote: > Hi guys, > > I just realize my code breaks with new versions of petsc-dev. I use > MatCreateMPIAIJWithSplitArrays() funtions and it works with this verison: > > Petsc Development HG revision: b27fc9457bdc002c5a5de2b07797ae1a93094370 > HG Date: Thu May 03 13:42:15 2012 -0500 > > but breaks with: > > Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 > HG Date: Thu Aug 16 08:37:18 2012 -0700 > > Here's the error code: > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: Column entry number 1 (actual colum 2) in row 3 is not > sorted! > Are your rows sorted? > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 > -0700 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./FVM on a arch-linu named mohammad-desktop by mohammad > Fri Aug 17 12:06:15 2012 > [0]PETSC ERROR: Libraries linked from > /home/mohammad/soft/petsc-dev_new/petsc-dev/arch-linux2-cxx-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 17 11:45:31 2012 > [0]PETSC ERROR: Configure options --with-clanguage=cxx > --with-mpi-dir=/usr/bin --download-f-blas-lapack=1 --download-hypre=1 > --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4268 in > /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatCreateMPIAIJWithSplitArrays() line 5731 in > /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: setLinearSystemMatrix() line 84 in > "unknowndirectory/"../../../CASL/lib/algebra/petscLinearSolver.cpp > > Is there anything changed regarding numbering of diagonal and > off-diagonal entries when using the said function between these two > versions? > > Thanks, > Mohammad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 17 14:20:11 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 17 Aug 2012 12:20:11 -0700 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: You mean columns within a row? Nope. On Fri, Aug 17, 2012 at 12:18 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 2:11 PM, Mohammad Mirzadeh wrote: > >> Hi guys, >> >> I just realize my code breaks with new versions of petsc-dev. I use >> MatCreateMPIAIJWithSplitArrays() funtions and it works with this verison: >> >> Petsc Development HG revision: b27fc9457bdc002c5a5de2b07797ae1a93094370 >> HG Date: Thu May 03 13:42:15 2012 -0500 >> >> but breaks with: >> >> Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 >> HG Date: Thu Aug 16 08:37:18 2012 -0700 >> >> Here's the error code: >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Argument out of range! >> [0]PETSC ERROR: Column entry number 1 (actual colum 2) in row 3 is not >> sorted! >> > > Are your rows sorted? > > >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Development HG revision: >> f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 >> -0700 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./FVM on a arch-linu named mohammad-desktop by mohammad >> Fri Aug 17 12:06:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /home/mohammad/soft/petsc-dev_new/petsc-dev/arch-linux2-cxx-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 17 11:45:31 2012 >> [0]PETSC ERROR: Configure options --with-clanguage=cxx >> --with-mpi-dir=/usr/bin --download-f-blas-lapack=1 --download-hypre=1 >> --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4268 in >> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatCreateMPIAIJWithSplitArrays() line 5731 in >> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: setLinearSystemMatrix() line 84 in >> "unknowndirectory/"../../../CASL/lib/algebra/petscLinearSolver.cpp >> >> Is there anything changed regarding numbering of diagonal and >> off-diagonal entries when using the said function between these two >> versions? >> >> Thanks, >> Mohammad >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 14:26:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 14:26:23 -0500 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 2:20 PM, Mohammad Mirzadeh wrote: > You mean columns within a row? Nope. > Okay, they are supposed to be. You can loop over your rows calling PetscSortInt() on each row separately. You didn't get the error before because debugging wasn't on, but you were also very lucky because many matrix kernels would have behaved incorrectly with unsorted data. > > > On Fri, Aug 17, 2012 at 12:18 PM, Jed Brown wrote: > >> On Fri, Aug 17, 2012 at 2:11 PM, Mohammad Mirzadeh wrote: >> >>> Hi guys, >>> >>> I just realize my code breaks with new versions of petsc-dev. I use >>> MatCreateMPIAIJWithSplitArrays() funtions and it works with this verison: >>> >>> Petsc Development HG revision: b27fc9457bdc002c5a5de2b07797ae1a93094370 >>> HG Date: Thu May 03 13:42:15 2012 -0500 >>> >>> but breaks with: >>> >>> Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 >>> HG Date: Thu Aug 16 08:37:18 2012 -0700 >>> >>> Here's the error code: >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Argument out of range! >>> [0]PETSC ERROR: Column entry number 1 (actual colum 2) in row 3 is not >>> sorted! >>> >> >> Are your rows sorted? >> >> >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: >>> f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 >>> -0700 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./FVM on a arch-linu named mohammad-desktop by mohammad >>> Fri Aug 17 12:06:15 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /home/mohammad/soft/petsc-dev_new/petsc-dev/arch-linux2-cxx-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Aug 17 11:45:31 2012 >>> [0]PETSC ERROR: Configure options --with-clanguage=cxx >>> --with-mpi-dir=/usr/bin --download-f-blas-lapack=1 --download-hypre=1 >>> --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4268 in >>> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/seq/aij.c >>> [0]PETSC ERROR: MatCreateMPIAIJWithSplitArrays() line 5731 in >>> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: setLinearSystemMatrix() line 84 in >>> "unknowndirectory/"../../../CASL/lib/algebra/petscLinearSolver.cpp >>> >>> Is there anything changed regarding numbering of diagonal and >>> off-diagonal entries when using the said function between these two >>> versions? >>> >>> Thanks, >>> Mohammad >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Fri Aug 17 14:27:20 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Fri, 17 Aug 2012 21:27:20 +0200 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: References: <502DFC8B.4080800@tu-dresden.de> <502E9619.3080807@tu-dresden.de> Message-ID: <502E9B18.7050809@tu-dresden.de> On Fri, Aug 17, 2012 at 3:10 AM, Thomas Witkowski > wrote: > >> I want to solve some (weakly) coupled system of equations of >> the following form: >> >> A B u >> . = ..... >> 0 C v >> >> >> so, C is the discrete Laplacian and A and B are some more >> complicated operators (I make use of linear finite elements). >> All boundary conditions are periodic, so the unknown v is >> determined only up to a constant. A and B contain both the >> identity operator, so u is fixed. Now I want to solve the >> system on the whole (there are reasons to do it in this way!) >> and I must provide information about the nullspace to the >> solver. When I am right, to provide the correct nullspace I >> must solve one equation with A. Is there any way in PETSc to >> circumvent the problem? >> >> >> If I understand you correctly, your null space vector is (0 I). I >> use the same null space for SNES ex62. > (0 I) cannot be an element of the null space, as multiplying it > with the matrix results in a non-zero vector. Or am I totally > wrong about null spaces of matrices? > > > Maybe you could as your question again. I am not understanding what > you want. > I want to solve the block triangular system as described above. My problem is, that it has a one dimensional null space, but I'm not able to define it. My question is: does anyone can give me an advice how to EITHER compute the null space explicitly OR how to solve the system in such a way that the null space is considered by the solver. The only constraint is that I cannot split the system of equations into two independent solve for both variables. I know that from this description its not clear why there is this constraint, but it would take too long to describe it. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 14:30:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 14:30:11 -0500 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: <502E9B18.7050809@tu-dresden.de> References: <502DFC8B.4080800@tu-dresden.de> <502E9619.3080807@tu-dresden.de> <502E9B18.7050809@tu-dresden.de> Message-ID: On Fri, Aug 17, 2012 at 2:27 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > I want to solve the block triangular system as described above. My problem > is, that it has a one dimensional null space, but I'm not able to define > it. My question is: does anyone can give me an advice how to EITHER compute > the null space explicitly OR how to solve the system in such a way that the > null space is considered by the solver. The only constraint is that I > cannot split the system of equations into two independent solve for both > variables. I know that from this description its not clear why there is > this constraint, but it would take too long to describe it. What are the operators involved? Why can't you determine the null space with a pencil and paper? Can you use PCFieldSplit in the solution process? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 17 14:39:42 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 14:39:42 -0500 Subject: [petsc-users] Nullspace for a coupled system of equations In-Reply-To: <502E9B18.7050809@tu-dresden.de> References: <502DFC8B.4080800@tu-dresden.de> <502E9619.3080807@tu-dresden.de> <502E9B18.7050809@tu-dresden.de> Message-ID: On Fri, Aug 17, 2012 at 2:27 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > On Fri, Aug 17, 2012 at 3:10 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > > I want to solve some (weakly) coupled system of equations of the >>> following form: >>> >>> A B u >>> . = ..... >>> 0 C v >>> >>> >>> so, C is the discrete Laplacian and A and B are some more complicated >>> operators (I make use of linear finite elements). All boundary conditions >>> are periodic, so the unknown v is determined only up to a constant. A and B >>> contain both the identity operator, so u is fixed. Now I want to solve the >>> system on the whole (there are reasons to do it in this way!) and I must >>> provide information about the nullspace to the solver. When I am right, to >>> provide the correct nullspace I must solve one equation with A. Is there >>> any way in PETSc to circumvent the problem? >> >> >> If I understand you correctly, your null space vector is (0 I). I use >> the same null space for SNES ex62. >> >> (0 I) cannot be an element of the null space, as multiplying it with the >> matrix results in a non-zero vector. Or am I totally wrong about null >> spaces of matrices? >> > > Maybe you could as your question again. I am not understanding what you > want. > > I want to solve the block triangular system as described above. My > problem is, that it has a one dimensional null space, but I'm not able to > define it. My question is: does anyone can give me an advice how to EITHER > compute the null space explicitly OR how to solve the system in such a way > that the null space is considered by the solver. The only constraint is > that I cannot split the system of equations into two independent solve for > both variables. I know that from this description its not clear why there > is this constraint, but it would take too long to describe it. > What is your evidence that it has a null space? Matt > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 17 14:42:02 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 17 Aug 2012 12:42:02 -0700 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: Thanks -- will do. Just curious, what's the reason for sorting? On Fri, Aug 17, 2012 at 12:26 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 2:20 PM, Mohammad Mirzadeh wrote: > >> You mean columns within a row? Nope. >> > > Okay, they are supposed to be. You can loop over your rows calling > PetscSortInt() on each row separately. > > You didn't get the error before because debugging wasn't on, but you were > also very lucky because many matrix kernels would have behaved incorrectly > with unsorted data. > > >> >> >> On Fri, Aug 17, 2012 at 12:18 PM, Jed Brown wrote: >> >>> On Fri, Aug 17, 2012 at 2:11 PM, Mohammad Mirzadeh wrote: >>> >>>> Hi guys, >>>> >>>> I just realize my code breaks with new versions of petsc-dev. I use >>>> MatCreateMPIAIJWithSplitArrays() funtions and it works with this verison: >>>> >>>> Petsc Development HG revision: b27fc9457bdc002c5a5de2b07797ae1a93094370 >>>> HG Date: Thu May 03 13:42:15 2012 -0500 >>>> >>>> but breaks with: >>>> >>>> Petsc Development HG revision: f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 >>>> HG Date: Thu Aug 16 08:37:18 2012 -0700 >>>> >>>> Here's the error code: >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Argument out of range! >>>> [0]PETSC ERROR: Column entry number 1 (actual colum 2) in row 3 is not >>>> sorted! >>>> >>> >>> Are your rows sorted? >>> >>> >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Development HG revision: >>>> f9c6cac2d69c724a2258d4e0dc2f51b0825aa875 HG Date: Thu Aug 16 08:37:18 2012 >>>> -0700 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./FVM on a arch-linu named mohammad-desktop by mohammad >>>> Fri Aug 17 12:06:15 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>> /home/mohammad/soft/petsc-dev_new/petsc-dev/arch-linux2-cxx-debug/lib >>>> [0]PETSC ERROR: Configure run at Fri Aug 17 11:45:31 2012 >>>> [0]PETSC ERROR: Configure options --with-clanguage=cxx >>>> --with-mpi-dir=/usr/bin --download-f-blas-lapack=1 --download-hypre=1 >>>> --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4268 in >>>> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/seq/aij.c >>>> [0]PETSC ERROR: MatCreateMPIAIJWithSplitArrays() line 5731 in >>>> /home/mohammad/soft/petsc-dev_new/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>> [0]PETSC ERROR: setLinearSystemMatrix() line 84 in >>>> "unknowndirectory/"../../../CASL/lib/algebra/petscLinearSolver.cpp >>>> >>>> Is there anything changed regarding numbering of diagonal and >>>> off-diagonal entries when using the said function between these two >>>> versions? >>>> >>>> Thanks, >>>> Mohammad >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 17 14:43:44 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 14:43:44 -0500 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: On Fri, Aug 17, 2012 at 2:42 PM, Mohammad Mirzadeh wrote: > Thanks -- will do. Just curious, what's the reason for sorting? Many operations need to be able to search or scan through entries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 17 14:44:37 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 17 Aug 2012 12:44:37 -0700 Subject: [petsc-users] issue with different versions of petsc-dev In-Reply-To: References: Message-ID: I see. Thanks again. On Fri, Aug 17, 2012 at 12:43 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 2:42 PM, Mohammad Mirzadeh wrote: > >> Thanks -- will do. Just curious, what's the reason for sorting? > > > Many operations need to be able to search or scan through entries. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Aug 17 15:06:38 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 16:06:38 -0400 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: Thanks for your help! John On Fri, Aug 17, 2012 at 2:59 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 1:48 PM, John Fettig wrote: >> >> On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: >> > Why do you want raw access? >> >> I'm trying to write my own RAP operator for AMG. >> >> > If you really need, you can either use MatMPIAIJGetSeqAIJ() or just >> > include >> > the private header. >> >> I presume you mean MatMPIAIJGetLocalMat. This isn't quite what I >> want. I don't want the local matrix, I want to scatter a vector >> according to the ghost values defined by the sparsity pattern of the >> parallel matrix. > > > PetscErrorCode MatMPIAIJGetSeqAIJ(Mat A,Mat *Ad,Mat *Ao,PetscInt **colmap); > > But if you are writing an RAP, you should just include > src/mat/impls/aij/mpi/mpiaij.h. You can set your implementation on an > existing matrix using MatShellSetOperation(A, MATOP_PTAP, > (void(*)(void))YourPtAPFunction). > >> >> Could you give me a little more help with using the latter? I see >> >> VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); >> (*a->A->ops->mult)(a->A,xx,yy); >> VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); >> (*a->B->ops->multadd)(a->B,a->lvec,yy,yy); >> >> >> So I guess a->Mvctx is what I want. How do I figure out the mapping >> of indices in a->lvec? > > > a->garray. MatPtAP is not an easy operation to write. You'll have to get > comfortable with diving into the source to understand what's happening. > > >> >> >> John > > From bsmith at mcs.anl.gov Fri Aug 17 15:34:06 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 15:34:06 -0500 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <502E4D0E.4080800@gfz-potsdam.de> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> Message-ID: <9BF071FB-5991-4821-A0B9-22808CA00E63@mcs.anl.gov> On Aug 17, 2012, at 8:54 AM, Alexander Grayver wrote: > On 17.08.2012 15:45, Jed Brown wrote: >> >> What I would like to do is very simple: every N iterations I want to take current solution vector x_i from KSP and calculate: >> x_i = x_i + c, >> where c is a correction vector. >> >> How do you want to compute c? > > I was planning to do it inside ShellPCApply. > The correction itself is calculated as a solution of Poisson equation with rhs being divergence of current density and current density is calculated based on x_i A preconditioner step can only be written in the form c = B(b - A x_i) where B is any linear operator and b is the right hand side and A is the original operator. So if your calculation of c of this form (note that you need not form B explicitly as a sparse matrix but it must be some linear operator. From you description above "with rhs being divergence of current density and current density is calculated based on x_i" I suspect it cannot be written in this form. The way to write this is KSPSetInitialGuessNonzero(kspmain); KSPSetTolerance(kspmain, set number of inner iterations you wish to use before the "filtering step" for j=1,?.. number outer iterations KSPSolve(kspmain,b,x) compute rhs via "rhs being divergence of current density and current density is calculated based on x_i" KSPSolve(ksppoisson,rhs,c) x = x + c endfor I don't think you can, or want to "munge" it into some kind of single complicated KSPSolve > This trick is often applied to suppress spurious solution arising during solution of Maxwell equations and it works very well. > >> For a general KSP, this is not allowed because the preconditioner must be constant. (You can do it anyway by checking the iteration number in your PCShell, but it won't converge.) You can play these games with a flexible method like FGMRES, for which you can also do an inner-outer, e.g. >> >> -ksp_type fgmres -pc_type composite -pc_composite_type multiplicative -pc_composite_pcs ksp,shell -sub_0_ksp_ksp_type tfqmr -sub_0_ksp_ksp_max_it 10 -sub_0_ksp_pc_type ilu > > Looks a bit tricky. The number of fgmres iterations then defines number of the cycles correction will be applied, doesn't it? > > > -- > Regards, > Alexander > From bsmith at mcs.anl.gov Fri Aug 17 15:39:56 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 15:39:56 -0500 Subject: [petsc-users] Segmentation fault when using matlab interface In-Reply-To: References: <17A78B9D13564547AC894B88C159674703413B@XMBX4.uibk.ac.at> <17A78B9D13564547AC894B88C1596747035169@XMBX4.uibk.ac.at> Message-ID: <112C8711-5D7C-4DFE-9511-73C3ACE1B377@mcs.anl.gov> On Aug 17, 2012, at 10:50 AM, Satish Balay wrote: > one issue with linking to 64bit matlab is: it internally links in with > 64int mlk/blas [packaged in matlab]. So I think we have to use this > mkl with petsc [along with -known-64-bit-blas-indices=1 option] > > But there should be no calls to blas with just petscinitialize/finalize. > > Perhaps its possible to run this in a debugger to determine the exact reason for SEGV? You can actually run MATLAB in the debugger (command line version). Look in script that starts up matlab to see what executable it actually uses and just start that up in the debugger. When it crashes you will be left in the debugger, now MATLAB uses lots of threads so you will need to switch between the threads (need to figure out/know a little bit of gdb to do this) until you find the thread that crashed and then you can see what call it was in. Similarly you could run the executable under valgrind to see if it is memory corruption causing the grief. It could be as simple as you are using the 64 bit version of MATLAB (which always uses 64 bit integers), did you build PETSc with the --with-64-bit-indices option? If not it will cause this crash; Send configure.log to petsc-maint at mcs.anl.gov Barry > > Satish > > > On Fri, 17 Aug 2012, De Groof, Vincent wrote: > >> Well, basically the simplest program I have the problem with is initializing and finalizing petsc. >> >> % Set environment variables. >> setenv('PETSC_DIR', '/usr/local/petsc-3.3-p2/') >> setenv('PETSC_ARCH','arch-linux2-c-debug') >> >> % Call PETSc. >> PetscInitialize(); >> PetscFinalize(); >> >> >> thanks. >> ________________________________ >> Von: De Groof, Vincent >> Gesendet: Freitag, 17. August 2012 10:25 >> An: petsc-users at mcs.anl.gov >> Betreff: Segmentation fault when using matlab interface >> >> Hello, >> >> I'm trying to use the Petsc objects directly from Matlab. However, when testing the examples in the /bin/matlab/classes/examples/tutorials-directory, I get a segmentation fault (core dump) error. Unfortunately, I don't get more information then this. After running some different situations, I know the following: >> I ran matlab both with and without desktop. The only error message I get is the segmentation fault. Sometimes it doesn't give the segmentation error after running the example. If so, it will cause a segmentation fault upon entering the next command in the matlab shell. If I quit matlab immediately after running an example without segmentation fault, my unix shell will hang. >> >> >> I know it's kind of vague, but this is all I know now. Any suggestions to look at to solve this problem? >> >> >> thanks. >> > From bsmith at mcs.anl.gov Fri Aug 17 16:00:56 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 16:00:56 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: <45BACBAA-8DC0-4228-A6A2-9A209DDCCE1C@mcs.anl.gov> MatGetCommunicationStructs() returns exactly back the VecScatter that gets all those ghost points. It is hardwired for MPIAIJ matrices; which is ok for you. PETSc developers need to fix this and either add MPIAIJ in the name or make it more general for other formats. Yucky stuff that should not have been exposed in exactly this way. Barry On Aug 17, 2012, at 1:33 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 1:05 PM, John Fettig wrote: > Is this the role of MatGetGhosts? > > That's the intent, but it looks like this function was only implemented for MPIBAIJ. > > The actual implementation of MatMult_MPI*AIJ is to start the scatter, apply the diagonal block, finish the scatter, and apply the off-diagonal part. > > Why do you want raw access? > > If you really need, you can either use MatMPIAIJGetSeqAIJ() or just include the private header. > > > John > > On Fri, Aug 17, 2012 at 2:01 PM, John Fettig wrote: > > Is there a scatter context available someplace that would allow me to > > gather all of the elements of a vector that correspond to the ones > > needed for a MatMult (MPIAIJ matrices)? When you do a MatMult, I > > presume it first scatters/gathers the ghost elements of the vector > > being multiplied, can I use this scatter for a vectors outside of the > > MatMult? > > > > Thanks, > > John > From bsmith at mcs.anl.gov Fri Aug 17 16:03:44 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 16:03:44 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: Message-ID: <54EED416-4BD7-468C-B665-ED6CDFCDD79F@mcs.anl.gov> On Aug 17, 2012, at 1:48 PM, John Fettig wrote: > On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: >> Why do you want raw access? > > I'm trying to write my own RAP operator for AMG. Why? Our's is not fast enough and you think you can make it better? You want to support an R that is not the transpose of P? Regardless why not start with the current source code we have and fork your own routine instead of starting from scratch? Doing a full efficient RAP from scratch is a major project. Barry > >> If you really need, you can either use MatMPIAIJGetSeqAIJ() or just include >> the private header. > > I presume you mean MatMPIAIJGetLocalMat. This isn't quite what I > want. I don't want the local matrix, I want to scatter a vector > according to the ghost values defined by the sparsity pattern of the > parallel matrix. > > Could you give me a little more help with using the latter? I see > > VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); > (*a->A->ops->mult)(a->A,xx,yy); > VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD); > (*a->B->ops->multadd)(a->B,a->lvec,yy,yy); > > So I guess a->Mvctx is what I want. How do I figure out the mapping > of indices in a->lvec? > > John From jedbrown at mcs.anl.gov Fri Aug 17 16:04:17 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 17 Aug 2012 16:04:17 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: <45BACBAA-8DC0-4228-A6A2-9A209DDCCE1C@mcs.anl.gov> References: <45BACBAA-8DC0-4228-A6A2-9A209DDCCE1C@mcs.anl.gov> Message-ID: On Fri, Aug 17, 2012 at 4:00 PM, Barry Smith wrote: > > MatGetCommunicationStructs() returns exactly back the VecScatter that gets > all those ghost points. > > It is hardwired for MPIAIJ matrices; which is ok for you. > > PETSc developers need to fix this and either add MPIAIJ in the name or > make it more general for other formats. Yucky stuff that should not have > been exposed in exactly this way. > Yikes, it's type signature depends on PETSC_USE_CTABLE. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 17 16:07:33 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 16:07:33 -0500 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: <45BACBAA-8DC0-4228-A6A2-9A209DDCCE1C@mcs.anl.gov> Message-ID: <1427B6F7-A7A6-40D7-AE1E-B3E1497A85D0@mcs.anl.gov> On Aug 17, 2012, at 4:04 PM, Jed Brown wrote: > On Fri, Aug 17, 2012 at 4:00 PM, Barry Smith wrote: > > MatGetCommunicationStructs() returns exactly back the VecScatter that gets all those ghost points. > > It is hardwired for MPIAIJ matrices; which is ok for you. > > PETSc developers need to fix this and either add MPIAIJ in the name or make it more general for other formats. Yucky stuff that should not have been exposed in exactly this way. > > Yikes, it's type signature depends on PETSC_USE_CTABLE. That's called showing a little too much flesh :-) From bsmith at mcs.anl.gov Fri Aug 17 16:14:48 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 16:14:48 -0500 Subject: [petsc-users] Strange behavior of MatLUFactorNumeric() In-Reply-To: <7237384C7A8F2642A8854B751619FA316F273A78@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA316F273822@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA316F273848@exchange.scsolutions.com> <903710B6-5D20-43AD-9156-8778CE38F9AF@mcs.anl.gov> <7237384C7A8F2642A8854B751619FA316F2738A5@exchange.scsolutions.com> <500F4234-3C1C-4F81-9AF5-3F30EC4D0A24@mcs.anl.gov> <7237384C7A8F2642A8854B751619FA316F273A78@exchange.scsolutions.com> Message-ID: <5F9239A7-74C9-418C-A220-A828FBA0EB6B@mcs.anl.gov> Those valgrind messages are not help, they are just issues with MPI implementation. If you see valgrind messages where the problems are inside PETSc or superlu_dist then those may be real problems (send us those valgrind messages if they exist to petsc-maint at mcs.anl.gov) You can track down the problem easily with a little debugger work. Consider running with -on_error_attach_debugger and then when it crashes in the debugger type where and look at the lines and variables where it crashed and you may find the problem. Basic usage of the debugger can save you endless amounts of pain. It is worth spending some hours getting comfortable in the debugger it will pay off big time. Barry On Aug 16, 2012, at 11:14 AM, Jinquan Zhong wrote: > Barry, > > ********************************************************************************************************************************* > Here is the typical info I got from Valgrind for a large matrix with order N=141081. Would they imply something wrong in the installation or the software setting? > ********************************************************************************************************************************* > > 1. Several circumstances where the following (Conditional jump or move) happens: > > ==28565== Conditional jump or move depends on uninitialised value(s) > ==28565== at 0x85A9590: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE4066AA: ibv_open_device (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79BD990: ring_rdma_open_hca (rdma_iba_priv.c:411) > ==28565== by 0x79CC6CD: rdma_setup_startup_ring (ring_startup.c:405) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > > > ==28565== Conditional jump or move depends on uninitialised value(s) > ==28565== at 0x79A59D1: calloc (mvapich_malloc.c:3719) > ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) > ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > > ==28565== Conditional jump or move depends on uninitialised value(s) > ==28565== at 0x4A08E54: memset (mc_replace_strmem.c:731) > ==28565== by 0x79A5BA1: calloc (mvapich_malloc.c:3825) > ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) > ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > > ==28565== Conditional jump or move depends on uninitialised value(s) > ==28565== at 0x4A08E79: memset (mc_replace_strmem.c:731) > ==28565== by 0x79A5BA1: calloc (mvapich_malloc.c:3825) > ==28565== by 0x85A99F2: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) > ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > > 2. > > ==28565== Syscall param write(buf) points to uninitialised byte(s) > ==28565== at 0x3FDF40E460: __write_nocancel (in /lib64/libpthread-2.12.so) > ==28565== by 0x3FDE404DFC: ibv_cmd_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x85AB742: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) > ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > ==28565== Address 0x7feffbae8 is on thread 1's stack > > > 3. > > ==28565== Use of uninitialised value of size 8 > ==28565== at 0x85A998F: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x85AB75A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==28565== by 0x3FDE408F21: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) > ==28565== by 0x79CC6A6: create_qp (ring_startup.c:223) > ==28565== by 0x79CC7DE: rdma_setup_startup_ring (ring_startup.c:434) > ==28565== by 0x79B98D2: MPIDI_CH3I_CM_Init (rdma_iba_init.c:1031) > ==28565== by 0x793C79C: MPIDI_CH3_Init (ch3_init.c:159) > ==28565== by 0x7996600: MPID_Init (mpid_init.c:288) > ==28565== by 0x799011E: MPIR_Init_thread (initthread.c:402) > ==28565== by 0x7990328: PMPI_Init_thread (initthread.c:569) > ==28565== by 0x481AE0: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:671) > ==28565== by 0x408ED9: main (hw.cpp:122) > > > > ******************************************* > The error message for rank 10 from PETSc is > ******************************************* > > [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [10]PETSC ERROR: INSTEAD the line number of the start of the function > [10]PETSC ERROR: is given. > [10]PETSC ERROR: --------------------- Error Message ------------------------------------ > [10]PETSC ERROR: Signal received! > [10]PETSC ERROR: ------------------------------------------------------------------------ > [10]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [10]PETSC ERROR: See docs/changes/index.html for recent updates. > [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [10]PETSC ERROR: See docs/index.html for manual pages. > [10]PETSC ERROR: ------------------------------------------------------------------------ > [10]PETSC ERROR: /nfs/06/com0488/programs/examples/testc/ZSOL on a arch-linu named n0685.ten.osc.edu by com0488 Wed Aug 15 21:00:31 2012 > [10]PETSC ERROR: Libraries linked from /nfs/07/com0489/petsc/petsc-3.3-p2/arch-linux2-cxx-debug/lib > [10]PETSC ERROR: Configure run at Wed Aug 15 13:55:29 2012 > [10]PETSC ERROR: Configure options --with-blas-lapack-lib="-L/usr/local/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -liomp5 -lm" --download-blacs --download-scalapack --with-mpi-dir=/usr/local/mvapich2/1.7-gnu --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex --with-precision=double --with-clanguage=cxx --with-fortran-kernels=generic --download-mumps --download-superlu_dist --download-parmetis --download-metis --with-fortran-interfaces > [10]PETSC ERROR: ------------------------------------------------------------------------ > [10]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [cli_10]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 > > > > Jinquan > > > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Tuesday, August 14, 2012 7:54 PM > To: PETSc users list > Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() > > > On Aug 14, 2012, at 6:05 PM, Jinquan Zhong wrote: > >> Barry, >> >> The machine I ran this program does not have valgrind. >> >> Another interesting observation is that when I ran the same three >> matrices using PETSc3.2. MatLUFactorNumeric() hanged up even on N=75, >> 2028 till I specified -mat_superlu_dist_colperm. However, >> MatLUFactorNumeric() didn't work for N=21180 either even I used >> >> -mat_superlu_dist_rowperm NATURAL -mat_superlu_dist_colperm NATURAL >> -mat_superlu_dist_parsymbfact YES >> >> I suspect that there is something incompatible in the factored matrix from superLU-dist to be used MatLUFactorNumeric() in PETSc3.2. Although PETSc 3.3 fixed this issue for matrix with small N, however, this issue relapsed for large N in PETSc3.3. > > It is using Superlu_dist for this factorization (and that version changed with PETSc 3.3) the problem is with Superlu_Dist not PETSc. valgrind will likely find an error in SuperLU_dist > > Barry > >> >> Jinquan >> >> >> -----Original Message----- >> From: petsc-users-bounces at mcs.anl.gov >> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith >> Sent: Tuesday, August 14, 2012 3:55 PM >> To: PETSc users list >> Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() >> >> >> Can you run with valgrind >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> >> >> On Aug 14, 2012, at 5:39 PM, Jinquan Zhong wrote: >> >>> Thanks, Matt. >>> >>> 1. Yes, I have checked the returned values from x obtained from >>> MatSolve(F,b,x) >>> >>> The norm error check for x is complete for N=75, 2028. >>> >>> 2. Good point, Matt. Here is the complete message for Rank 391. The others are similar to this one. >>> >>> >>> [391]PETSC ERROR: >>> --------------------------------------------------------------------- >>> - >>> -- [391]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>> Violation, probably memory access out of range [391]PETSC ERROR: Try >>> option -start_in_debugger or -on_error_attach_debugger [391]PETSC >>> ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[391]PETS >>> C >>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >>> find memory corruption errors [391]PETSC ERROR: likely location of >>> problem given in stack below [391]PETSC ERROR: --------------------- >>> Stack Frames ------------------------------------ >>> [391]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [391]PETSC ERROR: INSTEAD the line number of the start of the function >>> [391]PETSC ERROR: is given. >>> [391]PETSC ERROR: [391] MatLUFactorNumeric_SuperLU_DIST line 284 >>> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/impls/a >>> i j/mpi/superlu_dist/superlu_dist.c [391]PETSC ERROR: [391] >>> MatLUFactorNumeric line 2778 >>> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/src/mat/interfa >>> c e/matrix.c [391]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [391]PETSC ERROR: Signal received! >>> [391]PETSC ERROR: >>> --------------------------------------------------------------------- >>> - >>> -- [391]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>> 15:42:00 CDT 2012 [391]PETSC ERROR: See docs/changes/index.html for >>> recent updates. >>> [391]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [391]PETSC ERROR: See docs/index.html for manual pages. >>> [391]PETSC ERROR: >>> --------------------------------------------------------------------- >>> - >>> -- [391]PETSC ERROR: >>> /nfs/06/com0488/programs/examples/ZSOL0.2431/ZSOL >>> on a arch-linu named n0272.ten.osc.edu by com0488 Sun Aug 12 23:18:07 >>> 2012 [391]PETSC ERROR: Libraries linked from >>> /nfs/06/com0488/programs/libraries/PETSc/petsc-3.3-p2/arch-linux2-cxx >>> - debug/lib [391]PETSC ERROR: Configure run at Fri Aug 3 17:44:00 >>> 2012 [391]PETSC ERROR: Configure options >>> --with-blas-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/li >>> b >>> /librefblas.a >>> --with-lapack-lib=/nfs/06/com0488/programs/libraries/ScaLAPACK/2.0.1/ >>> l ib/libreflapack.a --download-blacs --download-scalapack >>> --with-mpi-dir=/usr/local/mvapich2/1.7-gnu >>> --with-mpiexec=/usr/local/bin/mpiexec --with-scalar-type=complex >>> --with-precision=double --with-clanguage=cxx >>> --with-fortran-kernels=generic --download-mumps >>> --download-superlu_dist --download-parmetis --download-metis >>> --with-fortran-interfaces[391]PETSC ERROR: >>> --------------------------------------------------------------------- >>> - >>> -- [391]PETSC ERROR: User provided function() line 0 in unknown >>> directory unknown file >>> [cli_391]: aborting job: >>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 391 >>> >>> >>> From: petsc-users-bounces at mcs.anl.gov >>> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >>> Sent: Tuesday, August 14, 2012 3:34 PM >>> To: PETSc users list >>> Subject: Re: [petsc-users] Strange behavior of MatLUFactorNumeric() >>> >>> On Tue, Aug 14, 2012 at 5:26 PM, Jinquan Zhong wrote: >>> Dear PETSc folks, >>> >>> I have a strange observation on using MatLUFactorNumeric() for dense matrices at different order N. Here is the situation I have: >>> >>> 1. I use ./src/mat/tests/ex137.c as an example to direct PETSc in selecting superLU-dist and mumps. The calling sequence is >>> >>> MatGetOrdering(A,...) >>> >>> MatGetFactor(A,...) >>> >>> MatLUFactorSymbolic(F, A,...) >>> >>> MatLUFactorNumeric(F, A,...) >>> >>> MatSolve(F,b,x) >>> >>> 2. I have three dense matrices A at three different dimensions: N=75, 2028 and 21180. >>> >>> 3. The calling sequence works for N=75 and 2028. But when N=21180, the program hanged up when calling MatLUFactorNumeric(...). Seemed to be a segmentation fault with the following error message: >>> >>> >>> >>> [1]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [1]PETSC ERROR: Signal received! >>> >>> ALWAYS send the entire error message. How can we tell anything from a small snippet? >>> >>> Since you have [1], this was run in parallel, so you need 3rd party >>> packages. But you do not seem to be checking return values. Check >>> them to make sure those packages are installed correctly. >>> >>> Matt >>> >>> Does anybody have similar experience on that? >>> >>> Thanks a lot! >>> >>> Jinquan >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > From bsmith at mcs.anl.gov Fri Aug 17 18:31:52 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Aug 2012 18:31:52 -0500 Subject: [petsc-users] Possible funding opportunity for PETSc based software products References: <42983BE3-C851-4545-BC25-4D50FCA4F9EB@mcs.anl.gov> Message-ID: <3589C679-002E-4858-87FE-D626C4A345B1@mcs.anl.gov> The US Department of Energy provides funding to small businesses to move advances from DOE funded research into industrial products. This year they have a focus on moving DOE software advances into industry. > > > > The DOE SBIR/STTR call came out on Monday (see http://www.er.doe.gov/grants/pdf/SC_FOA_0000760.pdf). It includes > > a. Turnkey HPC Solutions for Manufacturing and Engineering > c. Hardening of R&D Code for Industry Use > > under Increasing Adoption of HPC Modeling and Simulation in the Advanced Manufacturing and Engineering Industries. > > If you are developing/supporting products that utilize PETSc this is a great opportunity to have DOE fund some of the development work. Should you desire additional information please refer to the material above and to contact me directly. I am willing to provide letters of support and other help for you in preparing proposals. Good luck, Barry From john.fettig at gmail.com Fri Aug 17 19:38:52 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 17 Aug 2012 20:38:52 -0400 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: <54EED416-4BD7-468C-B665-ED6CDFCDD79F@mcs.anl.gov> References: <54EED416-4BD7-468C-B665-ED6CDFCDD79F@mcs.anl.gov> Message-ID: On Fri, Aug 17, 2012 at 5:03 PM, Barry Smith wrote: > > On Aug 17, 2012, at 1:48 PM, John Fettig wrote: > >> On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: >>> Why do you want raw access? >> >> I'm trying to write my own RAP operator for AMG. > > Why? Our's is not fast enough and you think you can make it better? You want to support an R that is not the transpose of P? Regardless why not start with the current source code we have and fork your own routine instead of starting from scratch? Doing a full efficient RAP from scratch is a major project. Yes, I'm having problems getting MatPtAP to work efficiently with my projection matrices. It is especially problematic in parallel, and I'm not really sure why since it seems to do ok for GAMG. I'm trying to code up a simple RAP currently, and I'll let you know how it goes. John From iamkyungjoo at gmail.com Fri Aug 17 19:51:09 2012 From: iamkyungjoo at gmail.com (Kyungjoo Kim) Date: Fri, 17 Aug 2012 19:51:09 -0500 Subject: [petsc-users] FEM, unassembled matrix based GMRES or CG method ? Message-ID: <0696AB99-897B-4A3B-8024-3CA7BE29136A@gmail.com> Dear PETSc experts: I am wondering if PETSc support Krylov methods without assembling FE-matrices. I googled and checked example codes but was not able to find out an answer for this Thank you. Kyungjoo From knepley at gmail.com Fri Aug 17 19:57:20 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Aug 2012 19:57:20 -0500 Subject: [petsc-users] FEM, unassembled matrix based GMRES or CG method ? In-Reply-To: <0696AB99-897B-4A3B-8024-3CA7BE29136A@gmail.com> References: <0696AB99-897B-4A3B-8024-3CA7BE29136A@gmail.com> Message-ID: On Fri, Aug 17, 2012 at 7:51 PM, Kyungjoo Kim wrote: > Dear PETSc experts: > > I am wondering if PETSc support Krylov methods without assembling > FE-matrices. > I googled and checked example codes but was not able to find out an answer > for this > Yes, use MATSHELL for your matrix. Matt > Thank you. > > Kyungjoo > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Aug 17 22:47:14 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 17 Aug 2012 23:47:14 -0400 Subject: [petsc-users] Scatter context for MatMult In-Reply-To: References: <54EED416-4BD7-468C-B665-ED6CDFCDD79F@mcs.anl.gov> Message-ID: <5037609D-1863-4206-A2B5-33C122C2C3E6@columbia.edu> On Aug 17, 2012, at 8:38 PM, John Fettig wrote: > On Fri, Aug 17, 2012 at 5:03 PM, Barry Smith wrote: >> >> On Aug 17, 2012, at 1:48 PM, John Fettig wrote: >> >>> On Fri, Aug 17, 2012 at 2:33 PM, Jed Brown wrote: >>>> Why do you want raw access? >>> >>> I'm trying to write my own RAP operator for AMG. >> >> Why? Our's is not fast enough and you think you can make it better? You want to support an R that is not the transpose of P? Regardless why not start with the current source code we have and fork your own routine instead of starting from scratch? Doing a full efficient RAP from scratch is a major project. > > Yes, I'm having problems getting MatPtAP to work efficiently with my > projection matrices. It is especially problematic in parallel, and > I'm not really sure why since it seems to do ok for GAMG. I'm trying > to code up a simple RAP currently, and I'll let you know how it goes. > As Barry says you are diving down a rabbit hole here unless you want to really get into it (which is fine). I think the GAMG projection ops that you were using were unsmoothed which are very sparse and so RAP does not do as much work as with most P (R'). Try the smoothing as I mentioned before and use this as your baseline for the speed of RAP. This smoothed operator probably looks more like your P than the unsmoothed one. > John > From agrayver at gfz-potsdam.de Sat Aug 18 02:40:23 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Sat, 18 Aug 2012 09:40:23 +0200 Subject: [petsc-users] Composite shell preconditiner In-Reply-To: <9BF071FB-5991-4821-A0B9-22808CA00E63@mcs.anl.gov> References: <502E3D8B.3060100@gfz-potsdam.de> <502E4311.2090503@gfz-potsdam.de> <502E48D7.4070205@gfz-potsdam.de> <502E4D0E.4080800@gfz-potsdam.de> <9BF071FB-5991-4821-A0B9-22808CA00E63@mcs.anl.gov> Message-ID: <502F46E7.6040706@gfz-potsdam.de> Barry, This was my original idea before I started trying pccomposite stuff. I will stick to it, thank you for pointing it out. On 17.08.2012 22:34, Barry Smith wrote: > On Aug 17, 2012, at 8:54 AM, Alexander Grayver wrote: > >> you want to compute c? >> I was planning to do it inside ShellPCApply. >> The correction itself is calculated as a solution of Poisson equation with rhs being divergence of current density and current density is calculated based on x_i > > A preconditioner step can only be written in the form c = B(b - A x_i) where B is any linear operator and b is the right hand side and A is the original operator. So if your calculation of c of this form (note that you need not form B explicitly as a sparse matrix but it must be some linear operator. From you description above "with rhs being divergence of current density and current density is calculated based on x_i" I suspect it cannot be written in this form. > > The way to write this is > > KSPSetInitialGuessNonzero(kspmain); > KSPSetTolerance(kspmain, set number of inner iterations you wish to use before the "filtering step" > for j=1,?.. number outer iterations > KSPSolve(kspmain,b,x) > compute rhs via "rhs being divergence of current density and current density is calculated based on x_i" > KSPSolve(ksppoisson,rhs,c) > x = x + c > endfor > > I don't think you can, or want to "munge" it into some kind of single complicated KSPSolve -- Regards, Alexander From fd.kong at siat.ac.cn Sat Aug 18 11:52:51 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Sun, 19 Aug 2012 00:52:51 +0800 Subject: [petsc-users] The "DMComplexDistribute" is time-consuming Message-ID: Hi developers, I use DMComplex represent the unstructured mesh. Everything is OK, but the function "DMComplexDistribute" is always time-consuming. I guess that it results from using the PetscSF. I don't know why we use this communication model. Could you please make some tests on the function? Or could we use other communication functions ? Regards, ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Aug 18 11:57:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 18 Aug 2012 11:57:41 -0500 Subject: [petsc-users] The "DMComplexDistribute" is time-consuming In-Reply-To: References: Message-ID: On Sat, Aug 18, 2012 at 11:52 AM, fdkong wrote: > Hi developers, > > I use DMComplex represent the unstructured mesh. Everything is OK, but the > function "DMComplexDistribute" is always time-consuming. I guess that it > results from using the PetscSF. I don't know why we use this communication > model. Could you please make some tests on the function? Or could we use > other communication functions ? > 1. Always send -log_summary 2. Distributing a mesh is a fairly expensive operation. 3. DMComplex is new and we are more interested in getting the right capability than in micro-tuning performance. Mesh distribution is usually not done frequently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Sat Aug 18 12:22:35 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Sun, 19 Aug 2012 01:22:35 +0800 Subject: [petsc-users] The "DMComplexDistribute" is time-consuming Message-ID: Hi Jed Brown, >> Hi developers, >> >> I use DMComplex represent the unstructured mesh. Everything is OK, but the >> function "DMComplexDistribute" is always time-consuming. I guess that it >> results from using the PetscSF. I don't know why we use this communication >> model. Could you please make some tests on the function? Or could we use >> other communication functions ? >> >1. Always send -log_summary >2. Distributing a mesh is a fairly expensive operation. >3. DMComplex is new and we are more interested in getting the right capability than in micro-tuning performance. Mesh distribution is usually not done frequently. ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_summary.txt Type: application/octet-stream Size: 14761 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sat Aug 18 12:27:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 18 Aug 2012 12:27:09 -0500 Subject: [petsc-users] The "DMComplexDistribute" is time-consuming In-Reply-To: References: Message-ID: ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Matt, why don't the mesh operations have events? I should ask myself the same question about PetscSF (and not forget to write the two-sided implementation). On Sat, Aug 18, 2012 at 12:22 PM, fdkong wrote: > Hi Jed Brown, > > >> Hi developers, > >> > >> I use DMComplex represent the unstructured mesh. Everything is OK, but > the > >> function "DMComplexDistribute" is always time-consuming. I guess that it > >> results from using the PetscSF. I don't know why we use this > communication > >> model. Could you please make some tests on the function? Or could we > use > >> other communication functions ? > >> > > >1. Always send -log_summary > > >2. Distributing a mesh is a fairly expensive operation. > > >3. DMComplex is new and we are more interested in getting the right > capability than in micro-tuning performance. Mesh distribution is usually > not done frequently. > > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Sun Aug 19 09:23:03 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Sun, 19 Aug 2012 08:23:03 -0600 Subject: [petsc-users] fortran NULL for VecScatterCreate In-Reply-To: References: Message-ID: Hello PETSC developers and users, In fortran the general PETSC_NULL is not valid. How can I pass a null argument iy in function VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) ? Thanks and regards, Panruo Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Aug 19 09:24:45 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 Aug 2012 09:24:45 -0500 Subject: [petsc-users] fortran NULL for VecScatterCreate In-Reply-To: References: Message-ID: On Sun, Aug 19, 2012 at 9:23 AM, Panruo Wu wrote: > > Hello PETSC developers and users, > > In fortran the general PETSC_NULL is not valid. How can I > pass a null argument iy in function > > VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) > > ? > > Thanks and regards, > PETSC_NULL_OBJECT for a Petsc object Matt > Panruo Wu > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Sun Aug 19 12:14:55 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Sun, 19 Aug 2012 11:14:55 -0600 Subject: [petsc-users] fortran NULL for VecScatterCreate In-Reply-To: References: Message-ID: Thank you Matt. I think it's worth adding to the petsc manual, along with the existing PETSC_NULL_{INTEGER, CHARACTER, DOUBLE SCALAR}. Panruo On Sun, Aug 19, 2012 at 8:24 AM, Matthew Knepley wrote: > On Sun, Aug 19, 2012 at 9:23 AM, Panruo Wu wrote: > >> >> Hello PETSC developers and users, >> >> In fortran the general PETSC_NULL is not valid. How can I >> pass a null argument iy in function >> >> VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) >> >> ? >> >> Thanks and regards, >> > > PETSC_NULL_OBJECT for a Petsc object > > Matt > > >> Panruo Wu >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Aug 19 13:19:46 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 19 Aug 2012 21:49:46 +0330 Subject: [petsc-users] How to use AD for unstructured grid Message-ID: Dear developers, I need to evaluate Jacobian matrix using an AD tool. I read from the Petsc manual that it can only work with structured grid via DMMG interface. Is there a possible way to use it for unstructured grid and set it for KSP structure? Thanks, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Aug 19 13:26:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 19 Aug 2012 13:26:19 -0500 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: On Sun, Aug 19, 2012 at 1:19 PM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear developers, > > I need to evaluate Jacobian matrix using an AD tool. I read from the Petsc > manual that it can only work with structured grid via DMMG interface. Is > there a possible way to use it for unstructured grid and set it for KSP > structure? > ADIC has not been supported for a long time so the "automatic" AD support in PETSc is no longer supported either. Unfortunately, the successors to ADIC, while more "capable", are much more involved to build and to use. You have always been able to use any AD tool internally to define the Jacobian (in assembled or unassembled form). -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Aug 19 13:36:47 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 19 Aug 2012 22:06:47 +0330 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: OK. Thanks. I should try it. On Sun, Aug 19, 2012 at 9:56 PM, Jed Brown wrote: > On Sun, Aug 19, 2012 at 1:19 PM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> Dear developers, >> >> I need to evaluate Jacobian matrix using an AD tool. I read from the >> Petsc manual that it can only work with structured grid via DMMG interface. >> Is there a possible way to use it for unstructured grid and set it for KSP >> structure? >> > > ADIC has not been supported for a long time so the "automatic" AD support > in PETSc is no longer supported either. Unfortunately, the successors to > ADIC, while more "capable", are much more involved to build and to use. > > You have always been able to use any AD tool internally to define the > Jacobian (in assembled or unassembled form). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Aug 20 00:24:13 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 20 Aug 2012 08:54:13 +0330 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: Dear Jed, Which AD tool you suggest in the case of fast evaluations? Operator Overlapping types like ADOL-C or source code transformation types like ADIC? In addition, I prefer the minimum possible change to the my code structure :-) Thanks a lot, BehZad On Sun, Aug 19, 2012 at 10:06 PM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > OK. Thanks. I should try it. > > > On Sun, Aug 19, 2012 at 9:56 PM, Jed Brown wrote: > >> On Sun, Aug 19, 2012 at 1:19 PM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> >>> Dear developers, >>> >>> I need to evaluate Jacobian matrix using an AD tool. I read from the >>> Petsc manual that it can only work with structured grid via DMMG interface. >>> Is there a possible way to use it for unstructured grid and set it for KSP >>> structure? >>> >> >> ADIC has not been supported for a long time so the "automatic" AD support >> in PETSc is no longer supported either. Unfortunately, the successors to >> ADIC, while more "capable", are much more involved to build and to use. >> >> You have always been able to use any AD tool internally to define the >> Jacobian (in assembled or unassembled form). >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 20 00:25:32 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 20 Aug 2012 00:25:32 -0500 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: This is a choice for you to make. Operator overloading is usually less intrusive to use, but also more limited in capability. On Mon, Aug 20, 2012 at 12:24 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear Jed, > > Which AD tool you suggest in the case of fast evaluations? Operator > Overlapping types like ADOL-C or source code transformation types like ADIC? > > In addition, I prefer the minimum possible change to the my code structure > :-) > > Thanks a lot, > BehZad > > > On Sun, Aug 19, 2012 at 10:06 PM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> OK. Thanks. I should try it. >> >> >> On Sun, Aug 19, 2012 at 9:56 PM, Jed Brown wrote: >> >>> On Sun, Aug 19, 2012 at 1:19 PM, behzad baghapour < >>> behzad.baghapour at gmail.com> wrote: >>> >>>> Dear developers, >>>> >>>> I need to evaluate Jacobian matrix using an AD tool. I read from the >>>> Petsc manual that it can only work with structured grid via DMMG interface. >>>> Is there a possible way to use it for unstructured grid and set it for KSP >>>> structure? >>>> >>> >>> ADIC has not been supported for a long time so the "automatic" AD >>> support in PETSc is no longer supported either. Unfortunately, the >>> successors to ADIC, while more "capable", are much more involved to build >>> and to use. >>> >>> You have always been able to use any AD tool internally to define the >>> Jacobian (in assembled or unassembled form). >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Aug 20 01:40:05 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 20 Aug 2012 10:10:05 +0330 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: OK. Does any of them consider a sparse function differentiation or one should define a dense matrix to recieve the result? If so, then is there any possiblity to face with memory allocation problem in Large scale simulations? On Mon, Aug 20, 2012 at 8:55 AM, Jed Brown wrote: > This is a choice for you to make. Operator overloading is usually less > intrusive to use, but also more limited in capability. > > > On Mon, Aug 20, 2012 at 12:24 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> Dear Jed, >> >> Which AD tool you suggest in the case of fast evaluations? Operator >> Overlapping types like ADOL-C or source code transformation types like ADIC? >> >> In addition, I prefer the minimum possible change to the my code >> structure :-) >> >> Thanks a lot, >> BehZad >> >> >> On Sun, Aug 19, 2012 at 10:06 PM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> >>> OK. Thanks. I should try it. >>> >>> >>> On Sun, Aug 19, 2012 at 9:56 PM, Jed Brown wrote: >>> >>>> On Sun, Aug 19, 2012 at 1:19 PM, behzad baghapour < >>>> behzad.baghapour at gmail.com> wrote: >>>> >>>>> Dear developers, >>>>> >>>>> I need to evaluate Jacobian matrix using an AD tool. I read from the >>>>> Petsc manual that it can only work with structured grid via DMMG interface. >>>>> Is there a possible way to use it for unstructured grid and set it for KSP >>>>> structure? >>>>> >>>> >>>> ADIC has not been supported for a long time so the "automatic" AD >>>> support in PETSc is no longer supported either. Unfortunately, the >>>> successors to ADIC, while more "capable", are much more involved to build >>>> and to use. >>>> >>>> You have always been able to use any AD tool internally to define the >>>> Jacobian (in assembled or unassembled form). >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Mon Aug 20 03:09:08 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 20 Aug 2012 10:09:08 +0200 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit Message-ID: <5031F0A4.5080407@tu-dresden.de> Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel preconditioner to use A_01 instead of A_10 to solve an upper triangular system? Thomas From gdiso at ustc.edu Mon Aug 20 03:54:59 2012 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 20 Aug 2012 16:54:59 +0800 (CST) Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: Message-ID: <1633659.290761345452899405.JavaMail.coremail@mail.ustc.edu> My code genius (2D/3D unstructured nonlinear solver for semiconductor simulation ) use adol-c for quite a long time. This AD tool is implemented by c++ operator overloading. Using an AD tool makes the development of Jacobian much much easier than handwriting. AD tool usually runs 2x slower than handwriting code, but build the Jacobian matrix is usually not a bottleneck for large scale nonlinear problems. You can find some information and source code here. https://github.com/cogenda/Genius-TCAD-Open The AD tool is just a head file include/math/adolc.h (stable) and its improved version include/math/adsmt.h (not fully tested). BTW,implement AD by operator overloading is also possible in python. Dear developers, I need to evaluate Jacobian matrix using an AD tool. I read from the Petsc manual that it can only work with structured grid via DMMG interface. Is there a possible way to use it for unstructured grid and set it for KSP structure? Thanks, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Aug 20 04:33:13 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 20 Aug 2012 13:03:13 +0330 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: <1633659.290761345452899405.JavaMail.coremail@mail.ustc.edu> References: <1633659.290761345452899405.JavaMail.coremail@mail.ustc.edu> Message-ID: Thank you. How do you handle the sparse matrix? On Mon, Aug 20, 2012 at 12:24 PM, Gong Ding wrote: > My code genius (2D/3D unstructured nonlinear solver for semiconductor > simulation ) use adol-c for quite a long time. This AD tool is implemented > by c++ operator overloading. Using an AD tool makes the development of > Jacobian much much easier than handwriting. AD tool usually runs 2x slower > than handwriting code, but build the Jacobian matrix is usually not a > bottleneck for large scale nonlinear problems. > > You can find some information and source code here. > https://github.com/cogenda/Genius-TCAD-Open > The AD tool is just a head file include/math/adolc.h (stable) and its > improved version include/math/adsmt.h (not fully tested). > > > BTW,implement AD by operator overloading is also possible in python. > > > > Dear developers, > > I need to evaluate Jacobian matrix using an AD tool. I read from the Petsc > manual that it can only work with structured grid via DMMG interface. Is > there a possible way to use it for unstructured grid and set it for KSP > structure? > > Thanks, > BehZad > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 20 06:39:27 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 06:39:27 -0500 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit In-Reply-To: <5031F0A4.5080407@tu-dresden.de> References: <5031F0A4.5080407@tu-dresden.de> Message-ID: On Mon, Aug 20, 2012 at 3:09 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel > preconditioner to use A_01 instead of A_10 to solve an upper triangular > system? It always uses A_01. What exactly do you want to do? Matt > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Mon Aug 20 07:07:02 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 20 Aug 2012 20:07:02 +0800 (CST) Subject: [petsc-users] The condition number of the ill-conditioned matrix Message-ID: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> Hello, I have a problem about the ill-conditioned matrix. I deal with the soil-water coupled problem in geotechnical engineering. It can be demonstrated that the stiffness matrix of soil-water coupled problem is ill-conditioned due to the coupled effects of large relative differences in stiffnesses and permeabilities of materials. But it should not be ill-conditioned for soil-water uncoupled problem, although there is no derivation in papers. In my three models, the condition numbers are 3.5595E+05, 1.8849E+06, 9.0206E+07 for soil-water coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water uncoupled ones, respectively. Therefore, in my opinion, the uncoupled models are also ill-conditioned due to the large condition numbers. However, when using GMRES, CG and CGS, the three models of uncoupled ones show rapid convergent rate while coupled models have bad convergence performance. It is a problem in my practical project in using PETSc although it seems a weaker connections with PETSc. So, anyone offers some clues? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Mon Aug 20 07:17:22 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Mon, 20 Aug 2012 08:17:22 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> Message-ID: <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> > Yes, just call DMSetDefaultSection(). However, note that you will have > everything sized > correctly, but handling topology and iteration for the residual calculation > would be completely > up to you. > > Matt It would be really great to define the initial data layout using IS to define the equation numbers of the splits, while letting the new DM stuff handle creating all the nested splits given command line options. Is something like this possible? Colin Quoting Matthew Knepley : > On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe wrote: > >> No, No, No. You do not have to write a DM implementation. >>> >>> You just have to define the data layout in a PetscSection and attach it to >>> the DM with DMSetDefaultSection. >>> >>> Matt >>> >> >> Does use of the PetscSection mean that it is neccesary to define a DMMesh? >> In other >> words is there a way to create the data layout for the physics without >> having to specify >> any information about the mesh? >> > > Yes, just call DMSetDefaultSection(). However, note that you will have > everything sized > correctly, but handling topology and iteration for the residual calculation > would be completely > up to you. > > Matt > > >> Thanks >> Colin >> >> >> >> Quoting Matthew Knepley : >> >> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>> wrote: >>> >>> >>>> >>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe >>> >wrote: >>>> >>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>> since >>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>> active >>>>> on a particular node. If I use block the equations I run out of memory >>>>> pretty quickly on my machine but can get to reasonable sized problems >>>>> with >>>>> the unblocked format. >>>>> >>>>> Matt, sorry I am not getting this, but I am still not sure how the DM >>>>> interface works. I can see in the function PCFieldSplitSetDefaults that >>>>> there is an initial call to DMCreateFieldDecomposition and subsequent >>>>> calls >>>>> to DMCreateSubDM based on the command line options. What I am missing is >>>>> how the first call to DMCreateFieldDecomposition is able to figure out >>>>> which equations belong to which field just from command line info such >>>>> as >>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>>>> >>>>> This might change slightly in the near future in petsc-dev to allow >>>> one to >>>> define splits using named fields. >>>> In any event, there has to be DM support to implement the decompositions >>>> over a particular mesh/problem over that mesh. >>>> With DMDA you can essentially get combinations of strided fields in each >>>> split. DMCOMPOSITE allows you >>>> to pull out combinations of the subproblems that were put in there to >>>> begin with. If you have your own mesh, you have to write >>>> a DM implementation around it to expose the available fields. >>>> >>>> >>> No, No, No. You do not have to write a DM implementation. >>> >>> You just have to define the data layout in a PetscSection and attach it to >>> the DM with DMSetDefaultSection. >>> >>> Matt >>> >>> >>> Dmitry. >>>> >>>> >>>>> Thanks >>>>> >>>>> Colin >>>>> >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>> >>>>>> wrote: >>>>>> >>>>>> Colin, >>>>>> >>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that are >>>>>>> zero >>>>>>> modulo a value of unity on the diagonal and the restrained value on >>>>>>> the >>>>>>> right-hand side). >>>>>>> >>>>>>> >>>>>>> However, this is not necessary with the DM interface. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> -sg >>>>>> >>>>>>> >>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>> >>>>>>> From what I can gather from the petsc-dev source it looks like the >>>>>>> >>>>>>>> commands in 4) will then generate the splits using strided blocks. >>>>>>>> The >>>>>>>> problem with that is the fortran code I am using (FEAP) uses petsc to >>>>>>>> assemble and solve the linear problem within its own nonlinear and >>>>>>>> time >>>>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>>>> boundary >>>>>>>> conditions applied to it so petsc only sees the active (unrestrained) >>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>> active >>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>> index >>>>>>>> sets defining the splits on my own. Will it still be possible to make >>>>>>>> use >>>>>>>> of the new DM functions in this case? >>>>>>>> >>>>>>>> FEAP website: >>>>>>>> http://www.ce.berkeley.edu/******projects/feap/ >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>> >>>>>>>> Quoting Matthew Knepley : >>>>>>>> >>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley >>>>>>> > >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>> cjm2176 at columbia.edu >>>>>>>>> >>>>>>>>> >wrote: >>>>>>>>>> >>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>>>> create >>>>>>>>>> >>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>>>> original >>>>>>>>>>> problem? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>>>> splits >>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>> >>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>> fields >>>>>>>>>> apart at once >>>>>>>>>> >>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>> like >>>>>>>>>> >>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I should have shown recursive: >>>>>>>>>> >>>>>>>>> >>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>> >>>>>>>>> will split 2,0 into two blocks. >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Colin McAuliffe >>>>>>>>>>> PhD Candidate >>>>>>>>>>> Columbia University >>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>> which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>> ------------------------------******----------------- >>>>>>> >>>>>>> Sanjay Govindjee, PhD, PE >>>>>>> Professor of Civil Engineering >>>>>>> Vice Chair for Academic Affairs >>>>>>> >>>>>>> 779 Davis Hall >>>>>>> Structural Engineering, Mechanics and Materials >>>>>>> Department of Civil Engineering >>>>>>> University of California >>>>>>> Berkeley, CA 94720-1710 >>>>>>> >>>>>>> Voice: +1 510 642 6060 >>>>>>> FAX: +1 510 643 5264 >>>>>>> s_g at berkeley.edu >>>>>>> http://www.ce.berkeley.edu/~******sanjay >>>>>>> >>>>>>> >< >>>>>>> http://www.ce.berkeley.edu/~****sanjay >>>>>>> >>>>>>> > >>>>>>> > >>>>>>> ------------------------------******----------------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From thomas.witkowski at tu-dresden.de Mon Aug 20 07:18:27 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 20 Aug 2012 14:18:27 +0200 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit In-Reply-To: References: <5031F0A4.5080407@tu-dresden.de> Message-ID: <50322B13.8090004@tu-dresden.de> Am 20.08.2012 13:39, schrieb Matthew Knepley: > On Mon, Aug 20, 2012 at 3:09 AM, Thomas Witkowski > > wrote: > > Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel > preconditioner to use A_01 instead of A_10 to solve an upper > triangular system? > > > It always uses A_01. What exactly do you want to do? In this case the documentation should be changed (see p. 86) Thomas > > Matt > > > Thomas > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 20 07:19:37 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 07:19:37 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> Message-ID: On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe wrote: > Yes, just call DMSetDefaultSection(). However, note that you will have >> everything sized >> correctly, but handling topology and iteration for the residual >> calculation >> would be completely >> up to you. >> >> Matt >> > > It would be really great to define the initial data layout using IS to > define the equation numbers of the splits, while letting the new DM stuff > handle creating all the nested splits given command line options. Is > something like this possible? > This is just what I say above. Is something not working? Matt > Colin > > Quoting Matthew Knepley : > > On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe > >wrote: >> >> No, No, No. You do not have to write a DM implementation. >>> >>>> >>>> You just have to define the data layout in a PetscSection and attach it >>>> to >>>> the DM with DMSetDefaultSection. >>>> >>>> Matt >>>> >>>> >>> Does use of the PetscSection mean that it is neccesary to define a >>> DMMesh? >>> In other >>> words is there a way to create the data layout for the physics without >>> having to specify >>> any information about the mesh? >>> >>> >> Yes, just call DMSetDefaultSection(). However, note that you will have >> everything sized >> correctly, but handling topology and iteration for the residual >> calculation >> would be completely >> up to you. >> >> Matt >> >> >> Thanks >>> Colin >>> >>> >>> >>> Quoting Matthew Knepley : >>> >>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>> >>>> wrote: >>>> >>>> >>>> >>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>> cjm2176 at columbia.edu >>>>> >wrote: >>>>> >>>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>> >>>>>> since >>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>>> active >>>>>> on a particular node. If I use block the equations I run out of memory >>>>>> pretty quickly on my machine but can get to reasonable sized problems >>>>>> with >>>>>> the unblocked format. >>>>>> >>>>>> Matt, sorry I am not getting this, but I am still not sure how the DM >>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>> that >>>>>> there is an initial call to DMCreateFieldDecomposition and subsequent >>>>>> calls >>>>>> to DMCreateSubDM based on the command line options. What I am missing >>>>>> is >>>>>> how the first call to DMCreateFieldDecomposition is able to figure out >>>>>> which equations belong to which field just from command line info such >>>>>> as >>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>>>>> >>>>>> This might change slightly in the near future in petsc-dev to allow >>>>>> >>>>> one to >>>>> define splits using named fields. >>>>> In any event, there has to be DM support to implement the >>>>> decompositions >>>>> over a particular mesh/problem over that mesh. >>>>> With DMDA you can essentially get combinations of strided fields in >>>>> each >>>>> split. DMCOMPOSITE allows you >>>>> to pull out combinations of the subproblems that were put in there to >>>>> begin with. If you have your own mesh, you have to write >>>>> a DM implementation around it to expose the available fields. >>>>> >>>>> >>>>> No, No, No. You do not have to write a DM implementation. >>>> >>>> You just have to define the data layout in a PetscSection and attach it >>>> to >>>> the DM with DMSetDefaultSection. >>>> >>>> Matt >>>> >>>> >>>> Dmitry. >>>> >>>>> >>>>> >>>>> Thanks >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>>> >>>>>> wrote: >>>>>>> >>>>>>> Colin, >>>>>>> >>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that are >>>>>>>> zero >>>>>>>> modulo a value of unity on the diagonal and the restrained value on >>>>>>>> the >>>>>>>> right-hand side). >>>>>>>> >>>>>>>> >>>>>>>> However, this is not necessary with the DM interface. >>>>>>>> >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> -sg >>>>>>> >>>>>>> >>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>> >>>>>>>> From what I can gather from the petsc-dev source it looks like the >>>>>>>> >>>>>>>> commands in 4) will then generate the splits using strided blocks. >>>>>>>>> The >>>>>>>>> problem with that is the fortran code I am using (FEAP) uses petsc >>>>>>>>> to >>>>>>>>> assemble and solve the linear problem within its own nonlinear and >>>>>>>>> time >>>>>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>>>>> boundary >>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>> (unrestrained) >>>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>>> active >>>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>>> index >>>>>>>>> sets defining the splits on my own. Will it still be possible to >>>>>>>>> make >>>>>>>>> use >>>>>>>>> of the new DM functions in this case? >>>>>>>>> >>>>>>>>> FEAP website: >>>>>>>>> http://www.ce.berkeley.edu/********projects/feap/ >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>> Quoting Matthew Knepley : >>>>>>>>> >>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>> knepley at gmail.com >>>>>>>>> > >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>> >>>>>>>>>> >wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>>>>> create >>>>>>>>>>> >>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>> >>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>>>>> original >>>>>>>>>>>> problem? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>>>>> splits >>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>> >>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>>> fields >>>>>>>>>>> apart at once >>>>>>>>>>> >>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>>> like >>>>>>>>>>> >>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I should have shown recursive: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>> >>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Colin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>> PhD Candidate >>>>>>>>>>>> Columbia University >>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>> which >>>>>>>>>>> their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>> which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> ------------------------------********----------------- >>>>>>>> >>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>> Professor of Civil Engineering >>>>>>>> Vice Chair for Academic Affairs >>>>>>>> >>>>>>>> 779 Davis Hall >>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>> Department of Civil Engineering >>>>>>>> University of California >>>>>>>> Berkeley, CA 94720-1710 >>>>>>>> >>>>>>>> Voice: +1 510 642 6060 >>>>>>>> FAX: +1 510 643 5264 >>>>>>>> s_g at berkeley.edu >>>>>>>> http://www.ce.berkeley.edu/~********sanjay >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> > >>>>>>>> >< >>>>>>>> http://www.ce.berkeley.edu/~******sanjay >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>> berkeley.edu/~sanjay > >>>>>>>> > >>>>>>>> > >>>>>>>> ------------------------------********----------------- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Colin McAuliffe >>>>>> PhD Candidate >>>>>> Columbia University >>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 20 07:29:24 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 07:29:24 -0500 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit In-Reply-To: <50322B13.8090004@tu-dresden.de> References: <5031F0A4.5080407@tu-dresden.de> <50322B13.8090004@tu-dresden.de> Message-ID: On Mon, Aug 20, 2012 at 7:18 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Am 20.08.2012 13:39, schrieb Matthew Knepley: > > On Mon, Aug 20, 2012 at 3:09 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel >> preconditioner to use A_01 instead of A_10 to solve an upper triangular >> system? > > > It always uses A_01. What exactly do you want to do? > > In this case the documentation should be changed (see p. 86) > I did not understand your question. The correct answer is: You can define fields in any order. Matt > Thomas > > > Matt > > >> >> Thomas >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 20 07:30:40 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 20 Aug 2012 07:30:40 -0500 Subject: [petsc-users] How to use AD for unstructured grid In-Reply-To: References: <1633659.290761345452899405.JavaMail.coremail@mail.ustc.edu> Message-ID: On Mon, Aug 20, 2012 at 4:33 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Thank you. How do you handle the sparse matrix? Use AD locally, at the element or quadrature level. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 20 07:39:16 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 20 Aug 2012 07:39:16 -0500 Subject: [petsc-users] The condition number of the ill-conditioned matrix In-Reply-To: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> References: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> Message-ID: On Mon, Aug 20, 2012 at 7:07 AM, w_ang_temp wrote: > Hello, > I have a problem about the ill-conditioned matrix. I deal with the > soil-water coupled problem in > geotechnical engineering. It can be demonstrated that the stiffness matrix > of soil-water coupled problem is > ill-conditioned due to the coupled effects of large relative differences > in stiffnesses and permeabilities > of materials. But it should not be ill-conditioned for soil-water > uncoupled problem, although there is no derivation > in papers. > In my three models, the condition numbers are 3.5595E+05, 1.8849E+06, > 9.0206E+07 for soil-water > coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water > uncoupled ones, respectively. > Therefore, in my opinion, the uncoupled models are also ill-conditioned > due to the large condition > numbers. > However, when using GMRES, CG and CGS, the three models of uncoupled > ones show rapid convergent rate > while coupled models have bad convergence performance. > This indicates that the spectrum of the simpler model has a small number of outliers. It could be as simple as boundary condition scaling or it could be more meaningful. The coupled model does not have that nice property, so convergence is slower. You should start with: 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled 2. find an effective preconditioner. PCFIELDSPLIT might work well, but you have to understand methods to know how to drive it. Look at the examples and read through the literature in your field to find something you want to implement, then ask here if you need suggestions for how to implement. (It can often be done using only command line options.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Mon Aug 20 07:47:12 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Mon, 20 Aug 2012 08:47:12 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> Message-ID: <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> I am just not seeing where in the section such an IS would go. As far as I can tell from the functions in vsection.c, the only item in the section which is an IS is bcIndices, defining points where bcs are applied. 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); 39 (*s)->atlasLayout.comm = comm; 40 (*s)->atlasLayout.pStart = -1; 41 (*s)->atlasLayout.pEnd = -1; 42 (*s)->atlasLayout.numDof = 1; 43 (*s)->atlasDof = PETSC_NULL; 44 (*s)->atlasOff = PETSC_NULL; 45 (*s)->bc = PETSC_NULL; 46 (*s)->bcIndices = PETSC_NULL; 47 (*s)->setup = PETSC_FALSE; 48 (*s)->numFields = 0; 49 (*s)->fieldNames = PETSC_NULL; 50 (*s)->field = PETSC_NULL; Quoting Matthew Knepley : > On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe wrote: > >> Yes, just call DMSetDefaultSection(). However, note that you will have >>> everything sized >>> correctly, but handling topology and iteration for the residual >>> calculation >>> would be completely >>> up to you. >>> >>> Matt >>> >> >> It would be really great to define the initial data layout using IS to >> define the equation numbers of the splits, while letting the new DM stuff >> handle creating all the nested splits given command line options. Is >> something like this possible? >> > > This is just what I say above. Is something not working? > > Matt > > >> Colin >> >> Quoting Matthew Knepley : >> >> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe >> >wrote: >>> >>> No, No, No. You do not have to write a DM implementation. >>>> >>>>> >>>>> You just have to define the data layout in a PetscSection and attach it >>>>> to >>>>> the DM with DMSetDefaultSection. >>>>> >>>>> Matt >>>>> >>>>> >>>> Does use of the PetscSection mean that it is neccesary to define a >>>> DMMesh? >>>> In other >>>> words is there a way to create the data layout for the physics without >>>> having to specify >>>> any information about the mesh? >>>> >>>> >>> Yes, just call DMSetDefaultSection(). However, note that you will have >>> everything sized >>> correctly, but handling topology and iteration for the residual >>> calculation >>> would be completely >>> up to you. >>> >>> Matt >>> >>> >>> Thanks >>>> Colin >>>> >>>> >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>> cjm2176 at columbia.edu >>>>>> >wrote: >>>>>> >>>>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>>> >>>>>>> since >>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>>>> active >>>>>>> on a particular node. If I use block the equations I run out of memory >>>>>>> pretty quickly on my machine but can get to reasonable sized problems >>>>>>> with >>>>>>> the unblocked format. >>>>>>> >>>>>>> Matt, sorry I am not getting this, but I am still not sure how the DM >>>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>>> that >>>>>>> there is an initial call to DMCreateFieldDecomposition and subsequent >>>>>>> calls >>>>>>> to DMCreateSubDM based on the command line options. What I am missing >>>>>>> is >>>>>>> how the first call to DMCreateFieldDecomposition is able to figure out >>>>>>> which equations belong to which field just from command line info such >>>>>>> as >>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 defined? >>>>>>> >>>>>>> This might change slightly in the near future in petsc-dev to allow >>>>>>> >>>>>> one to >>>>>> define splits using named fields. >>>>>> In any event, there has to be DM support to implement the >>>>>> decompositions >>>>>> over a particular mesh/problem over that mesh. >>>>>> With DMDA you can essentially get combinations of strided fields in >>>>>> each >>>>>> split. DMCOMPOSITE allows you >>>>>> to pull out combinations of the subproblems that were put in there to >>>>>> begin with. If you have your own mesh, you have to write >>>>>> a DM implementation around it to expose the available fields. >>>>>> >>>>>> >>>>>> No, No, No. You do not have to write a DM implementation. >>>>> >>>>> You just have to define the data layout in a PetscSection and attach it >>>>> to >>>>> the DM with DMSetDefaultSection. >>>>> >>>>> Matt >>>>> >>>>> >>>>> Dmitry. >>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> Quoting Matthew Knepley : >>>>>>> >>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> Colin, >>>>>>>> >>>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that are >>>>>>>>> zero >>>>>>>>> modulo a value of unity on the diagonal and the restrained value on >>>>>>>>> the >>>>>>>>> right-hand side). >>>>>>>>> >>>>>>>>> >>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>> >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> -sg >>>>>>>> >>>>>>>> >>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>> >>>>>>>>> From what I can gather from the petsc-dev source it looks like the >>>>>>>>> >>>>>>>>> commands in 4) will then generate the splits using strided blocks. >>>>>>>>>> The >>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses petsc >>>>>>>>>> to >>>>>>>>>> assemble and solve the linear problem within its own nonlinear and >>>>>>>>>> time >>>>>>>>>> stepping schemes. The linear problem that petsc solves already has >>>>>>>>>> boundary >>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>> (unrestrained) >>>>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>>>> active >>>>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>>>> index >>>>>>>>>> sets defining the splits on my own. Will it still be possible to >>>>>>>>>> make >>>>>>>>>> use >>>>>>>>>> of the new DM functions in this case? >>>>>>>>>> >>>>>>>>>> FEAP website: >>>>>>>>>> http://www.ce.berkeley.edu/********projects/feap/ >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>> >>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>> knepley at gmail.com >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>> >>>>>>>>>>> >wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now to >>>>>>>>>>>> create >>>>>>>>>>>> >>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>> >>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>> to create a new DM based from the new fields and the DM for the >>>>>>>>>>>>> original >>>>>>>>>>>>> problem? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to define >>>>>>>>>>>> splits >>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>> >>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>>>> fields >>>>>>>>>>>> apart at once >>>>>>>>>>>> >>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off any >>>>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>>>> like >>>>>>>>>>>> >>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>> >>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>> Columbia University >>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>> which >>>>>>>>>>>> their >>>>>>>>>>>> experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>> which >>>>>>>>>>> their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> ------------------------------********----------------- >>>>>>>>> >>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>> Professor of Civil Engineering >>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>> >>>>>>>>> 779 Davis Hall >>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>> Department of Civil Engineering >>>>>>>>> University of California >>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>> >>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>> s_g at berkeley.edu >>>>>>>>> http://www.ce.berkeley.edu/~********sanjay >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> > >>>>>>>>> >< >>>>>>>>> http://www.ce.berkeley.edu/~******sanjay >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>> berkeley.edu/~sanjay > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ------------------------------********----------------- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From bsmith at mcs.anl.gov Mon Aug 20 08:10:08 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Aug 2012 08:10:08 -0500 Subject: [petsc-users] The condition number of the ill-conditioned matrix In-Reply-To: References: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> Message-ID: <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> I think it is even simpler than this. Let your coupled matrix look like ( A C12 ) ( C21 B ) and the uncoupled be ( A 0 ) ( 0 B ) Now solving the uncoupled with GMRES, CG etc is actually solving at the same time two independent systems and the relative size of the eigenvalues of A and B don't really matter. Jed is right that the first thing you need to do is > 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled you want the A part and the B part in the Jacobian and function evaluations to have similar scales of values. BTW: the FAQ has one sentence "Change the units (nondimensionalization), boundary condition scaling, or formulation so that the Jacobian is better conditioned." How about putting some references there and some good examples on how to do this? What this means? This is hard shit and we dismiss it in one sentence like a kindergartener can do it. Barry On Aug 20, 2012, at 7:39 AM, Jed Brown wrote: > On Mon, Aug 20, 2012 at 7:07 AM, w_ang_temp wrote: > Hello, > I have a problem about the ill-conditioned matrix. I deal with the soil-water coupled problem in > geotechnical engineering. It can be demonstrated that the stiffness matrix of soil-water coupled problem is > ill-conditioned due to the coupled effects of large relative differences in stiffnesses and permeabilities > of materials. But it should not be ill-conditioned for soil-water uncoupled problem, although there is no derivation > in papers. > In my three models, the condition numbers are 3.5595E+05, 1.8849E+06, 9.0206E+07 for soil-water > coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water uncoupled ones, respectively. > Therefore, in my opinion, the uncoupled models are also ill-conditioned due to the large condition > numbers. > However, when using GMRES, CG and CGS, the three models of uncoupled ones show rapid convergent rate > while coupled models have bad convergence performance. > > This indicates that the spectrum of the simpler model has a small number of outliers. It could be as simple as boundary condition scaling or it could be more meaningful. The coupled model does not have that nice property, so convergence is slower. You should start with: > > 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled > > 2. find an effective preconditioner. PCFIELDSPLIT might work well, but you have to understand methods to know how to drive it. Look at the examples and read through the literature in your field to find something you want to implement, then ask here if you need suggestions for how to implement. (It can often be done using only command line options.) From bsmith at mcs.anl.gov Mon Aug 20 08:19:50 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Aug 2012 08:19:50 -0500 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit In-Reply-To: <50322B13.8090004@tu-dresden.de> References: <5031F0A4.5080407@tu-dresden.de> <50322B13.8090004@tu-dresden.de> Message-ID: <50DAABA8-D7FB-4B78-9D38-F5518B28729E@mcs.anl.gov> I think you are asking if you have A00 A01 0 A11 Can you use field split trivially to solve this upper block triangular system by doing "backward/reverse" block Gauss-Seidel? The very short answer is no 1) you can use -pc_fieldsplit_type symmetric_multiplicative but then it does the A00 solve twice unnecessarily 2) what Matt is saying you can define you 00 index set to be associated with the second block A11 then that flips the block rows and columns of the matrix and the regular multiplicative field split will work like the backward block GS on the original matrix. Barry Yes, we could add something like -pc_fieldsplit_type backward_multiplicative but as Matt points out that is not strictly needed. On Aug 20, 2012, at 7:18 AM, Thomas Witkowski wrote: > Am 20.08.2012 13:39, schrieb Matthew Knepley: >> On Mon, Aug 20, 2012 at 3:09 AM, Thomas Witkowski wrote: >> Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel preconditioner to use A_01 instead of A_10 to solve an upper triangular system? >> >> It always uses A_01. What exactly do you want to do? > In this case the documentation should be changed (see p. 86) > > Thomas > >> >> Matt >> >> >> Thomas >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From knepley at gmail.com Mon Aug 20 09:36:50 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 09:36:50 -0500 Subject: [petsc-users] The condition number of the ill-conditioned matrix In-Reply-To: <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> References: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> Message-ID: On Mon, Aug 20, 2012 at 8:10 AM, Barry Smith wrote: > > I think it is even simpler than this. Let your coupled matrix look > like > > ( A C12 ) > ( C21 B ) > > and the uncoupled be > > ( A 0 ) > ( 0 B ) > > > Now solving the uncoupled with GMRES, CG etc is actually solving at > the same time two independent systems and the relative size of the > eigenvalues of A and B don't really matter. > > Jed is right that the first thing you need to do is > > > 1. non-dimensionalize: scale your model to use some representative units > so that the solution and residual are well-scaled > > > you want the A part and the B part in the Jacobian and function > evaluations to have similar scales of values. > > > BTW: the FAQ has one sentence "Change the units (nondimensionalization), > boundary condition scaling, or formulation so that the Jacobian is better > conditioned." > How about putting some references there and some good examples on how to > do this? What this means? This is hard shit and we dismiss it in one > sentence like a kindergartener can do it. http://en.wikipedia.org/wiki/Buckingham_%CF%80_theorem Matt > > Barry > > > On Aug 20, 2012, at 7:39 AM, Jed Brown wrote: > > > On Mon, Aug 20, 2012 at 7:07 AM, w_ang_temp wrote: > > Hello, > > I have a problem about the ill-conditioned matrix. I deal with the > soil-water coupled problem in > > geotechnical engineering. It can be demonstrated that the stiffness > matrix of soil-water coupled problem is > > ill-conditioned due to the coupled effects of large relative differences > in stiffnesses and permeabilities > > of materials. But it should not be ill-conditioned for soil-water > uncoupled problem, although there is no derivation > > in papers. > > In my three models, the condition numbers are 3.5595E+05, > 1.8849E+06, 9.0206E+07 for soil-water > > coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water > uncoupled ones, respectively. > > Therefore, in my opinion, the uncoupled models are also ill-conditioned > due to the large condition > > numbers. > > However, when using GMRES, CG and CGS, the three models of uncoupled > ones show rapid convergent rate > > while coupled models have bad convergence performance. > > > > This indicates that the spectrum of the simpler model has a small number > of outliers. It could be as simple as boundary condition scaling or it > could be more meaningful. The coupled model does not have that nice > property, so convergence is slower. You should start with: > > > > 1. non-dimensionalize: scale your model to use some representative units > so that the solution and residual are well-scaled > > > > 2. find an effective preconditioner. PCFIELDSPLIT might work well, but > you have to understand methods to know how to drive it. Look at the > examples and read through the literature in your field to find something > you want to implement, then ask here if you need suggestions for how to > implement. (It can often be done using only command line options.) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From tisaac at ices.utexas.edu Mon Aug 20 09:49:18 2012 From: tisaac at ices.utexas.edu (Toby Isaac) Date: Mon, 20 Aug 2012 09:49:18 -0500 Subject: [petsc-users] =?utf-8?q?Can_a_MatShell_that_operates_on_ghosted_v?= =?utf-8?q?ectors_work_in_place=3F?= Message-ID: <8f59fbd1698ac078f8bd5347282c1ae0@ices.utexas.edu> I have finite element code, developed outside of PETSc thus far, with matrix-free operators. I've written the operators so that they expect ghosted input vectors and return ghosted output vectors. I know that I can write a MatShell that creates a temporary ghosted versions of the input Vec and output Vec and calls my operators on these. What I want to know is if there is any way to work in place, i.e. write a MatShell that expects ghosted inputs/outputs. This would also require KSP methods to work with ghosted vectors and only use the locally-owned variables in the dot products. Thanks, Toby From jedbrown at mcs.anl.gov Mon Aug 20 09:53:39 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 20 Aug 2012 09:53:39 -0500 Subject: [petsc-users] Can a MatShell that operates on ghosted vectors work in place? In-Reply-To: <8f59fbd1698ac078f8bd5347282c1ae0@ices.utexas.edu> References: <8f59fbd1698ac078f8bd5347282c1ae0@ices.utexas.edu> Message-ID: On Mon, Aug 20, 2012 at 9:49 AM, Toby Isaac wrote: > > I have finite element code, developed outside of PETSc thus far, with > matrix-free operators. I've written the operators so that they expect > ghosted input vectors and return ghosted output vectors. > > I know that I can write a MatShell that creates a temporary ghosted > versions of the input Vec and output Vec and calls my operators on these. > What I want to know is if there is any way to work in place, i.e. write a > MatShell that expects ghosted inputs/outputs. This would also require KSP > methods to work with ghosted vectors and only use the locally-owned > variables in the dot products. > You can write one that expects VecGhost (or your own vector type). To do that, you usually want to implement MatGetVecs() and make sure solvers get passed ghosted initial vectors. Then MatMult_YourShell() uses VecGhostUpdateBegin()/End, VecGhostGetLocalForm(), your local evaluation into the residual vector, VecGhostRestoreLocalForm(), and VecGhostUpdateBegin() to finish assembling the residual vector. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Mon Aug 20 09:56:12 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 20 Aug 2012 16:56:12 +0200 Subject: [petsc-users] Solving upper triangular system with pc fieldsplit In-Reply-To: <50DAABA8-D7FB-4B78-9D38-F5518B28729E@mcs.anl.gov> References: <5031F0A4.5080407@tu-dresden.de> <50322B13.8090004@tu-dresden.de> <50DAABA8-D7FB-4B78-9D38-F5518B28729E@mcs.anl.gov> Message-ID: <5032500C.1030508@tu-dresden.de> Am 20.08.2012 15:19, schrieb Barry Smith: > I think you are asking if you have > > A00 A01 > 0 A11 > > Can you use field split trivially to solve this upper block triangular system by doing "backward/reverse" block Gauss-Seidel? yes. > > The very short answer is no > > 1) you can use -pc_fieldsplit_type symmetric_multiplicative but then it does the A00 solve twice unnecessarily Okay, this is at least a fast possibility to check fore some solver options, and its absolutely fine for my case. Thank you, Thomas > > 2) what Matt is saying you can define you 00 index set to be associated with the second block A11 then that flips the block rows and columns of the matrix and > the regular multiplicative field split will work like the backward block GS on the original matrix. > > Barry > > > Yes, we could add something like -pc_fieldsplit_type backward_multiplicative but as Matt points out that is not strictly needed. > > > > > > > On Aug 20, 2012, at 7:18 AM, Thomas Witkowski wrote: > >> Am 20.08.2012 13:39, schrieb Matthew Knepley: >>> On Mon, Aug 20, 2012 at 3:09 AM, Thomas Witkowski wrote: >>> Is it possible to change the PCFIELDSPLIT / block Gauss-Seidel preconditioner to use A_01 instead of A_10 to solve an upper triangular system? >>> >>> It always uses A_01. What exactly do you want to do? >> In this case the documentation should be changed (see p. 86) >> >> Thomas >> >>> Matt >>> >>> >>> Thomas >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener From w_ang_temp at 163.com Mon Aug 20 09:58:51 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 20 Aug 2012 22:58:51 +0800 (CST) Subject: [petsc-users] The condition number of the ill-conditioned matrix In-Reply-To: <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> References: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> Message-ID: <40ff6c29.141c0.139448b1c2b.Coremail.w_ang_temp@163.com> Thanks. I agree with you. Maybe I need to get the representative units. Also, maybe it is not reasonable to evaluate the performance in my project just under the condition number. At 2012-08-20 21:10:08,"Barry Smith" wrote: > > I think it is even simpler than this. Let your coupled matrix look like > > ( A C12 ) > ( C21 B ) > > and the uncoupled be > > ( A 0 ) > ( 0 B ) > > > Now solving the uncoupled with GMRES, CG etc is actually solving at the same time two independent systems and the relative size of the >eigenvalues of A and B don't really matter. > > Jed is right that the first thing you need to do is > >> 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled > > >you want the A part and the B part in the Jacobian and function evaluations to have similar scales of values. > > >BTW: the FAQ has one sentence "Change the units (nondimensionalization), boundary condition scaling, or formulation so that the Jacobian is bet>ter conditioned." >How about putting some references there and some good examples on how to do this? What this means? This is hard shit and we dismiss it in on>e sentence like a kindergartener can do it. > > Barry > > >On Aug 20, 2012, at 7:39 AM, Jed Brown wrote: > >> On Mon, Aug 20, 2012 at 7:07 AM, w_ang_temp wrote: >> Hello, >> I have a problem about the ill-conditioned matrix. I deal with the soil-water coupled problem in >> geotechnical engineering. It can be demonstrated that the stiffness matrix of soil-water coupled problem is >> ill-conditioned due to the coupled effects of large relative differences in stiffnesses and permeabilities >> of materials. But it should not be ill-conditioned for soil-water uncoupled problem, although there is no derivation >> in papers. >> In my three models, the condition numbers are 3.5595E+05, 1.8849E+06, 9.0206E+07 for soil-water >> coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water uncoupled ones, respectively. >> Therefore, in my opinion, the uncoupled models are also ill-conditioned due to the large condition >> numbers. >> However, when using GMRES, CG and CGS, the three models of uncoupled ones show rapid convergent rate >> while coupled models have bad convergence performance. >> >> This indicates that the spectrum of the simpler model has a small number of outliers. It could be as simple as boundary condition scaling or it cou>>ld be more meaningful. The coupled model does not have that nice property, so convergence is slower. You should start with: >> >> 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled >> >> 2. find an effective preconditioner. PCFIELDSPLIT might work well, but you have to understand methods to know how to drive it. Look at the exa>>mples and read through the literature in your field to find something you want to implement, then ask here if you need suggestions for how to imple>>ment. (It can often be done using only command line options.) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 20 10:03:56 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Aug 2012 10:03:56 -0500 Subject: [petsc-users] The condition number of the ill-conditioned matrix In-Reply-To: References: <79c4c58d.18ff5.13943edd1b5.Coremail.w_ang_temp@163.com> <8F078DCE-1A20-4DD9-A131-0E86EBE7D4CC@mcs.anl.gov> Message-ID: On Aug 20, 2012, at 9:36 AM, Matthew Knepley wrote: > On Mon, Aug 20, 2012 at 8:10 AM, Barry Smith wrote: > > I think it is even simpler than this. Let your coupled matrix look like > > ( A C12 ) > ( C21 B ) > > and the uncoupled be > > ( A 0 ) > ( 0 B ) > > > Now solving the uncoupled with GMRES, CG etc is actually solving at the same time two independent systems and the relative size of the > eigenvalues of A and B don't really matter. > > Jed is right that the first thing you need to do is > > > 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled > > > you want the A part and the B part in the Jacobian and function evaluations to have similar scales of values. > > > BTW: the FAQ has one sentence "Change the units (nondimensionalization), boundary condition scaling, or formulation so that the Jacobian is better conditioned." > How about putting some references there and some good examples on how to do this? What this means? This is hard shit and we dismiss it in one sentence like a kindergartener can do it. > > http://en.wikipedia.org/wiki/Buckingham_%CF%80_theorem > > Matt Thanks, added to FAQ > > > Barry > > > On Aug 20, 2012, at 7:39 AM, Jed Brown wrote: > > > On Mon, Aug 20, 2012 at 7:07 AM, w_ang_temp wrote: > > Hello, > > I have a problem about the ill-conditioned matrix. I deal with the soil-water coupled problem in > > geotechnical engineering. It can be demonstrated that the stiffness matrix of soil-water coupled problem is > > ill-conditioned due to the coupled effects of large relative differences in stiffnesses and permeabilities > > of materials. But it should not be ill-conditioned for soil-water uncoupled problem, although there is no derivation > > in papers. > > In my three models, the condition numbers are 3.5595E+05, 1.8849E+06, 9.0206E+07 for soil-water > > coupled models and 5.2885E+04, 3.3826 E+05, 3.4565E+05 for soil-water uncoupled ones, respectively. > > Therefore, in my opinion, the uncoupled models are also ill-conditioned due to the large condition > > numbers. > > However, when using GMRES, CG and CGS, the three models of uncoupled ones show rapid convergent rate > > while coupled models have bad convergence performance. > > > > This indicates that the spectrum of the simpler model has a small number of outliers. It could be as simple as boundary condition scaling or it could be more meaningful. The coupled model does not have that nice property, so convergence is slower. You should start with: > > > > 1. non-dimensionalize: scale your model to use some representative units so that the solution and residual are well-scaled > > > > 2. find an effective preconditioner. PCFIELDSPLIT might work well, but you have to understand methods to know how to drive it. Look at the examples and read through the literature in your field to find something you want to implement, then ask here if you need suggestions for how to implement. (It can often be done using only command line options.) > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Mon Aug 20 10:09:15 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 10:09:15 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> Message-ID: On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe wrote: > I am just not seeing where in the section such an IS would go. As far as I > can tell from the functions in vsection.c, the only item in the section > which is an IS is bcIndices, defining points where bcs are applied. > If you have no structure other than dof, you can make a Section where the points just number dofs. Then you just run through the ints p in your IS can call PetscSectionSetDof(section, p, 1) Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you want, and FS will use it. Matt > 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); > 39 (*s)->atlasLayout.comm = comm; > 40 (*s)->atlasLayout.pStart = -1; > 41 (*s)->atlasLayout.pEnd = -1; > 42 (*s)->atlasLayout.numDof = 1; > 43 (*s)->atlasDof = PETSC_NULL; > 44 (*s)->atlasOff = PETSC_NULL; > 45 (*s)->bc = PETSC_NULL; > 46 (*s)->bcIndices = PETSC_NULL; > 47 (*s)->setup = PETSC_FALSE; > 48 (*s)->numFields = 0; > 49 (*s)->fieldNames = PETSC_NULL; > 50 (*s)->field = PETSC_NULL; > > Quoting Matthew Knepley : > > On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe > >wrote: >> >> Yes, just call DMSetDefaultSection(). However, note that you will have >>> >>>> everything sized >>>> correctly, but handling topology and iteration for the residual >>>> calculation >>>> would be completely >>>> up to you. >>>> >>>> Matt >>>> >>>> >>> It would be really great to define the initial data layout using IS to >>> define the equation numbers of the splits, while letting the new DM stuff >>> handle creating all the nested splits given command line options. Is >>> something like this possible? >>> >>> >> This is just what I say above. Is something not working? >> >> Matt >> >> >> Colin >>> >>> Quoting Matthew Knepley : >>> >>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe >> >>>> >wrote: >>>> >>>> No, No, No. You do not have to write a DM implementation. >>>> >>>>> >>>>> >>>>>> You just have to define the data layout in a PetscSection and attach >>>>>> it >>>>>> to >>>>>> the DM with DMSetDefaultSection. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>> DMMesh? >>>>> In other >>>>> words is there a way to create the data layout for the physics without >>>>> having to specify >>>>> any information about the mesh? >>>>> >>>>> >>>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>> everything sized >>>> correctly, but handling topology and iteration for the residual >>>> calculation >>>> would be completely >>>> up to you. >>>> >>>> Matt >>>> >>>> >>>> Thanks >>>> >>>>> Colin >>>>> >>>>> >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>>>> >>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>> cjm2176 at columbia.edu >>>>>>> >wrote: >>>>>>> >>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>>>> >>>>>>> since >>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>>>>> active >>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>> memory >>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>> problems >>>>>>>> with >>>>>>>> the unblocked format. >>>>>>>> >>>>>>>> Matt, sorry I am not getting this, but I am still not sure how the >>>>>>>> DM >>>>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>>>> that >>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>> subsequent >>>>>>>> calls >>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>> missing >>>>>>>> is >>>>>>>> how the first call to DMCreateFieldDecomposition is able to figure >>>>>>>> out >>>>>>>> which equations belong to which field just from command line info >>>>>>>> such >>>>>>>> as >>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>> defined? >>>>>>>> >>>>>>>> This might change slightly in the near future in petsc-dev to allow >>>>>>>> >>>>>>>> one to >>>>>>> define splits using named fields. >>>>>>> In any event, there has to be DM support to implement the >>>>>>> decompositions >>>>>>> over a particular mesh/problem over that mesh. >>>>>>> With DMDA you can essentially get combinations of strided fields in >>>>>>> each >>>>>>> split. DMCOMPOSITE allows you >>>>>>> to pull out combinations of the subproblems that were put in there to >>>>>>> begin with. If you have your own mesh, you have to write >>>>>>> a DM implementation around it to expose the available fields. >>>>>>> >>>>>>> >>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>> >>>>>> >>>>>> You just have to define the data layout in a PetscSection and attach >>>>>> it >>>>>> to >>>>>> the DM with DMSetDefaultSection. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Dmitry. >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>> >>>>>>>> Quoting Matthew Knepley : >>>>>>>> >>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>>>> > >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Colin, >>>>>>>>> >>>>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>>> >>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>> are >>>>>>>>>> zero >>>>>>>>>> modulo a value of unity on the diagonal and the restrained value >>>>>>>>>> on >>>>>>>>>> the >>>>>>>>>> right-hand side). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>> -sg >>>>>>>>> >>>>>>>>> >>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>> >>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>> the >>>>>>>>>> >>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>> blocks. >>>>>>>>>> >>>>>>>>>>> The >>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>> petsc >>>>>>>>>>> to >>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>> and >>>>>>>>>>> time >>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>> has >>>>>>>>>>> boundary >>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>> (unrestrained) >>>>>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>>>>> active >>>>>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>>>>> index >>>>>>>>>>> sets defining the splits on my own. Will it still be possible to >>>>>>>>>>> make >>>>>>>>>>> use >>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>> >>>>>>>>>>> FEAP website: >>>>>>>>>>> http://www.ce.berkeley.edu/**********projects/feap/ >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Colin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>> knepley at gmail.com >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>> >>>>>>>>>>>> >wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now >>>>>>>>>>>>> to >>>>>>>>>>>>> create >>>>>>>>>>>>> >>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>> >>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>> the >>>>>>>>>>>>>> original >>>>>>>>>>>>>> problem? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>> define >>>>>>>>>>>>> splits >>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>> >>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>>>>> fields >>>>>>>>>>>>> apart at once >>>>>>>>>>>>> >>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>> any >>>>>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>>>>> like >>>>>>>>>>>>> >>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>> >>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>> which >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>> which >>>>>>>>>>>> their >>>>>>>>>>>> experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>> Professor of Civil Engineering >>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>> >>>>>>>>>> 779 Davis Hall >>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>> Department of Civil Engineering >>>>>>>>>> University of California >>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>> >>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>> s_g at berkeley.edu >>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >< >>>>>>>>>> http://www.ce.berkeley.edu/~********sanjay >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>> which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> Colin McAuliffe >>>>>>>> PhD Candidate >>>>>>>> Columbia University >>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 20 10:12:51 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Aug 2012 10:12:51 -0500 Subject: [petsc-users] Can a MatShell that operates on ghosted vectors work in place? In-Reply-To: References: <8f59fbd1698ac078f8bd5347282c1ae0@ices.utexas.edu> Message-ID: On Mon, Aug 20, 2012 at 9:53 AM, Jed Brown wrote: > On Mon, Aug 20, 2012 at 9:49 AM, Toby Isaac wrote: > >> >> I have finite element code, developed outside of PETSc thus far, with >> matrix-free operators. I've written the operators so that they expect >> ghosted input vectors and return ghosted output vectors. >> >> I know that I can write a MatShell that creates a temporary ghosted >> versions of the input Vec and output Vec and calls my operators on these. >> What I want to know is if there is any way to work in place, i.e. write a >> MatShell that expects ghosted inputs/outputs. This would also require KSP >> methods to work with ghosted vectors and only use the locally-owned >> variables in the dot products. >> > > You can write one that expects VecGhost (or your own vector type). To do > that, you usually want to implement MatGetVecs() and make sure solvers get > passed ghosted initial vectors. Then MatMult_YourShell() uses > VecGhostUpdateBegin()/End, VecGhostGetLocalForm(), your local evaluation > into the residual vector, VecGhostRestoreLocalForm(), and > VecGhostUpdateBegin() to finish assembling the residual vector. > Personally, I think this is quite cumbersome compared to the Local/Global vector strategy we use in DM, and does not have any real advantages. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 20 10:15:31 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 20 Aug 2012 10:15:31 -0500 Subject: [petsc-users] Can a MatShell that operates on ghosted vectors work in place? In-Reply-To: References: <8f59fbd1698ac078f8bd5347282c1ae0@ices.utexas.edu> Message-ID: On Mon, Aug 20, 2012 at 10:12 AM, Matthew Knepley wrote: > Personally, I think this is quite cumbersome compared to the Local/Global > vector strategy we use in DM, and > does not have any real advantages. > For example, storage for the ghost values is allocated for every vector duplicated from a VecGhost (e.g. the entire Krylov basis). This isn't an issue if you have large subdomains, but for smallish subdomains using methods with a fat stencil, that can double the storage needed for the Krylov basis. -------------- next part -------------- An HTML attachment was scrubbed... URL: From armiuswu at gmail.com Sat Aug 18 23:57:21 2012 From: armiuswu at gmail.com (Panruo Wu) Date: Sat, 18 Aug 2012 22:57:21 -0600 Subject: [petsc-users] fortran NULL for VecScatterCreate Message-ID: Hi PETSC developers and users, In fortran the general PETSC_NULL is not valid. How can I pass a null argument iy in function VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) ? Thanks and regards, Panruo Wu From bsmith at mcs.anl.gov Mon Aug 20 10:32:08 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Aug 2012 10:32:08 -0500 Subject: [petsc-users] fortran NULL for VecScatterCreate In-Reply-To: References: Message-ID: <32400853-3003-4140-AE85-7D633BEA8EE1@mcs.anl.gov> PETSC_NULL_OBJECT On Aug 18, 2012, at 11:57 PM, Panruo Wu wrote: > Hi PETSC developers and users, > > In fortran the general PETSC_NULL is not valid. How can I > pass a null argument iy in function > > VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) > > ? > > Thanks and regards, > > Panruo Wu From cjm2176 at columbia.edu Tue Aug 21 07:51:51 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Tue, 21 Aug 2012 08:51:51 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> Message-ID: <20120821085151.izr649usgg8c444s@cubmail.cc.columbia.edu> Ok thanks I'll give this a shot. I was originally trying to do one point with all dof, but one dof per point sounds more strait foreword. Quoting Matthew Knepley : > On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe wrote: > >> I am just not seeing where in the section such an IS would go. As far as I >> can tell from the functions in vsection.c, the only item in the section >> which is an IS is bcIndices, defining points where bcs are applied. >> > > If you have no structure other than dof, you can make a Section where the > points > just number dofs. Then you just run through the ints p in your IS can call > > PetscSectionSetDof(section, p, 1) > > Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you > want, > and FS will use it. > > Matt > > >> 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); >> 39 (*s)->atlasLayout.comm = comm; >> 40 (*s)->atlasLayout.pStart = -1; >> 41 (*s)->atlasLayout.pEnd = -1; >> 42 (*s)->atlasLayout.numDof = 1; >> 43 (*s)->atlasDof = PETSC_NULL; >> 44 (*s)->atlasOff = PETSC_NULL; >> 45 (*s)->bc = PETSC_NULL; >> 46 (*s)->bcIndices = PETSC_NULL; >> 47 (*s)->setup = PETSC_FALSE; >> 48 (*s)->numFields = 0; >> 49 (*s)->fieldNames = PETSC_NULL; >> 50 (*s)->field = PETSC_NULL; >> >> Quoting Matthew Knepley : >> >> On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe >> >wrote: >>> >>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>> >>>>> everything sized >>>>> correctly, but handling topology and iteration for the residual >>>>> calculation >>>>> would be completely >>>>> up to you. >>>>> >>>>> Matt >>>>> >>>>> >>>> It would be really great to define the initial data layout using IS to >>>> define the equation numbers of the splits, while letting the new DM stuff >>>> handle creating all the nested splits given command line options. Is >>>> something like this possible? >>>> >>>> >>> This is just what I say above. Is something not working? >>> >>> Matt >>> >>> >>> Colin >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe >>> >>>>> >wrote: >>>>> >>>>> No, No, No. You do not have to write a DM implementation. >>>>> >>>>>> >>>>>> >>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>> it >>>>>>> to >>>>>>> the DM with DMSetDefaultSection. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>>> DMMesh? >>>>>> In other >>>>>> words is there a way to create the data layout for the physics without >>>>>> having to specify >>>>>> any information about the mesh? >>>>>> >>>>>> >>>>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>>> everything sized >>>>> correctly, but handling topology and iteration for the residual >>>>> calculation >>>>> would be completely >>>>> up to you. >>>>> >>>>> Matt >>>>> >>>>> >>>>> Thanks >>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>>>>> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>>> cjm2176 at columbia.edu >>>>>>>> >wrote: >>>>>>>> >>>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>>>>> >>>>>>>> since >>>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>>>>>> active >>>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>>> memory >>>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>>> problems >>>>>>>>> with >>>>>>>>> the unblocked format. >>>>>>>>> >>>>>>>>> Matt, sorry I am not getting this, but I am still not sure how the >>>>>>>>> DM >>>>>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>>>>> that >>>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>>> subsequent >>>>>>>>> calls >>>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>>> missing >>>>>>>>> is >>>>>>>>> how the first call to DMCreateFieldDecomposition is able to figure >>>>>>>>> out >>>>>>>>> which equations belong to which field just from command line info >>>>>>>>> such >>>>>>>>> as >>>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>>> defined? >>>>>>>>> >>>>>>>>> This might change slightly in the near future in petsc-dev to allow >>>>>>>>> >>>>>>>>> one to >>>>>>>> define splits using named fields. >>>>>>>> In any event, there has to be DM support to implement the >>>>>>>> decompositions >>>>>>>> over a particular mesh/problem over that mesh. >>>>>>>> With DMDA you can essentially get combinations of strided fields in >>>>>>>> each >>>>>>>> split. DMCOMPOSITE allows you >>>>>>>> to pull out combinations of the subproblems that were put in there to >>>>>>>> begin with. If you have your own mesh, you have to write >>>>>>>> a DM implementation around it to expose the available fields. >>>>>>>> >>>>>>>> >>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>> >>>>>>> >>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>> it >>>>>>> to >>>>>>> the DM with DMSetDefaultSection. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Dmitry. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>> Quoting Matthew Knepley : >>>>>>>>> >>>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>>>>> > >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Colin, >>>>>>>>>> >>>>>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>>>> >>>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>>> are >>>>>>>>>>> zero >>>>>>>>>>> modulo a value of unity on the diagonal and the restrained value >>>>>>>>>>> on >>>>>>>>>>> the >>>>>>>>>>> right-hand side). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -sg >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>>> >>>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>>> blocks. >>>>>>>>>>> >>>>>>>>>>>> The >>>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>>> petsc >>>>>>>>>>>> to >>>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>>> and >>>>>>>>>>>> time >>>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>>> has >>>>>>>>>>>> boundary >>>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>>> (unrestrained) >>>>>>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>>>>>> active >>>>>>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>>>>>> index >>>>>>>>>>>> sets defining the splits on my own. Will it still be possible to >>>>>>>>>>>> make >>>>>>>>>>>> use >>>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>>> >>>>>>>>>>>> FEAP website: >>>>>>>>>>>> http://www.ce.berkeley.edu/**********projects/feap/ >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>>> knepley at gmail.com >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>>> >>>>>>>>>>>>> >wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now >>>>>>>>>>>>>> to >>>>>>>>>>>>>> create >>>>>>>>>>>>>> >>>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>>> >>>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> original >>>>>>>>>>>>>>> problem? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>>> define >>>>>>>>>>>>>> splits >>>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>>>>>> fields >>>>>>>>>>>>>> apart at once >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>>> any >>>>>>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>>>>>> like >>>>>>>>>>>>>> >>>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>>> >>>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>>> which >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>> which >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>>> Professor of Civil Engineering >>>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>>> >>>>>>>>>>> 779 Davis Hall >>>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>>> Department of Civil Engineering >>>>>>>>>>> University of California >>>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>>> >>>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >< >>>>>>>>>>> http://www.ce.berkeley.edu/~********sanjay >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>>> >>>>>>>>>>> >> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>> which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Colin McAuliffe >>>>>>>>> PhD Candidate >>>>>>>>> Columbia University >>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Colin McAuliffe >>>>>> PhD Candidate >>>>>> Columbia University >>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From hzhang at mcs.anl.gov Tue Aug 21 11:14:23 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 21 Aug 2012 11:14:23 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <502A25C2.2070003@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> Message-ID: Alexander : Testing your matrix A.dat using petsc-3.3/src/ksp/ksp/examples/tutorials/ex10.c, I find that A is numerically singular: a) petsc sequential lu: ./ex10 -f0 A.dat -ksp_type preonly -pc_type lu -rhs 0 [0]PETSC ERROR: Detected zero pivot in LU factorization: see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot! [0]PETSC ERROR: Zero pivot row 48 value 0 tolerance 2.22045e-14! b) mumps lu with np=8: mpiexec -n 8 ./ex10 -f0 A.dat -ksp_type preonly -pc_type lu -rhs 0 -pc_factor_mat_solver_package mumps ... [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=1 ... c) superlu_dist lu with np=8: mpiexec -n 8 ./ex10 -f0 A.dat -ksp_type preonly -pc_type lu -rhs 0 -pc_factor_mat_solver_package superlu_dist Number of iterations = 1 Residual norm 0.0103982 mumps cholesky with np=8: mpiexec -n 8 ./ex10 -f0 A.dat -ksp_type preonly -pc_type cholesky -rhs 0 -pc_factor_mat_solver_package mumps Number of iterations = 1 Residual norm 122744 With such large residual norms, the computed solution cannot be trusted :-( d) using -ksp_type gmres with pc=ilu: zero pivot pc=jacobi: iteration stagnates .. I'm convinced that matrix A is corrupted or obtained from an incorrect numerical model. Also note, your matrix A has size of approx. 60k, which is too large for sequential direct solver. The error on MatMatSolve() when calling ' -pc_type cholesky -pc_factor_mat_solver_package mumps' is a bug in our petsc-mumps interface. I've pushed a fix to petsc-3.3 http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/8badc49a596e Thanks for your report. Hong On 13.08.2012 19:01, Hong Zhang wrote: > >> Alexander : >> I need a simple code from you to reproduce the error for debugging. >> > > Hong, > > Attached you will find test.c that should reproduce this behavior with > petsc-3.3-p2. > Just run it with 1 and 2 procs as following: > ./mpirun -n 1 test -ksp_type preonly -pc_type cholesky > -pc_factor_mat_solver_package mumps -ksp_view > > Here is matrix I tested it on (15 MB): https://dl.dropbox.com/u/** > 60982984/A.dat > > > If mumps' solver is activated, then you should get error (sequential and >> parallel) >> [0]PETSC ERROR: No support for this operation for this object type! >> [0]PETSC ERROR: MatMatSolve_MUMPS() is not implemented yet! >> >> If petsc's direct solver is activated, then you should not be able to run >> it in parallel. >> > > Please note, that it works correctly for nproc > 1 since 1 year or so! > So I don't quite understand why there should be error thrown. > I just occasionally decided to test something in sequential version and > this thing happened. > > > >> Please check ~petsc/src/mat/examples/tests/**ex125.c >> to see if you used same calling procedure as this example. >> Note: I fixed a bug in this example and patched petsc-3.3. >> The update copy of ex125.c is attached for your convenience. >> >> Hong >> >> > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Tue Aug 21 11:25:36 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 21 Aug 2012 18:25:36 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> Message-ID: <5033B680.1030109@gfz-potsdam.de> On 21.08.2012 18:14, Hong Zhang wrote: > Alexander : > > Testing your matrix A.dat > using petsc-3.3/src/ksp/ksp/examples/tutorials/ex10.c, > I find that A is numerically singular: Hong, Sorry, but I forgot to mention that this is double complex symmetric matrix. Did you load and treat it like that? > I'm convinced that matrix A is corrupted or obtained from an incorrect > numerical model. > Also note, your matrix A has size of approx. 60k, which is too large > for sequential direct solver. MUMPS takes only several minutes and 6 GB of memory to factorize it. This factorization gives residual on the order of 10e-12 and solution is indeed correct. Nevertheless, you're right, there is numerical null-space in this matrix since it comes from the discretization of equation that contains curl curl operator, but practically this case is not really the worst one. > > The error on MatMatSolve() when calling > ' -pc_type cholesky -pc_factor_mat_solver_package mumps' > is a bug in our petsc-mumps interface. I've pushed a fix to petsc-3.3 > http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/8badc49a596e > > Thanks for your report. Thank you for the patch. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Aug 21 11:31:45 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 21 Aug 2012 11:31:45 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <5033B680.1030109@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> Message-ID: Alexander: > > > Sorry, but I forgot to mention that this is double complex symmetric > matrix. Did you load and treat it like that? > I'm not aware it, thus I used real precision :-( > > > I'm convinced that matrix A is corrupted or obtained from an incorrect > numerical model. > Also note, your matrix A has size of approx. 60k, which is too large for > sequential direct solver. > > > MUMPS takes only several minutes and 6 GB of memory to factorize it. > This factorization gives residual on the order of 10e-12 and solution is > indeed correct. > > Nevertheless, you're right, there is numerical null-space in this matrix > since it comes > from the discretization of equation that contains curl curl operator, but > practically this > case is not really the worst one. > > > The error on MatMatSolve() when calling > ' -pc_type cholesky -pc_factor_mat_solver_package mumps' > is a bug in our petsc-mumps interface. I've pushed a fix to petsc-3.3 > http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/8badc49a596e > > Thanks for your report. > > > Thank you for the patch. > > -- > Regards, > Alexander > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 21 11:32:46 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Aug 2012 11:32:46 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <5033B680.1030109@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 11:25 AM, Alexander Grayver wrote: > On 21.08.2012 18:14, Hong Zhang wrote: > > Alexander : > > Testing your matrix A.dat > using petsc-3.3/src/ksp/ksp/examples/tutorials/ex10.c, > I find that A is numerically singular: > > > Hong, > > Sorry, but I forgot to mention that this is double complex symmetric > matrix. Did you load and treat it like that? > > I'm convinced that matrix A is corrupted or obtained from an incorrect > numerical model. > Also note, your matrix A has size of approx. 60k, which is too large for > sequential direct solver. > > > MUMPS takes only several minutes and 6 GB of memory to factorize it. > This factorization gives residual on the order of 10e-12 and solution is > indeed correct. > > Nevertheless, you're right, there is numerical null-space in this matrix > since it comes > from the discretization of equation that contains curl curl operator, but > practically this > case is not really the worst one. > This makes no sense whatsoever. How can you LU factor a matrix that has a null space? Matt > > The error on MatMatSolve() when calling > ' -pc_type cholesky -pc_factor_mat_solver_package mumps' > is a bug in our petsc-mumps interface. I've pushed a fix to petsc-3.3 > http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/8badc49a596e > > Thanks for your report. > > > Thank you for the patch. > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Tue Aug 21 11:42:02 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 21 Aug 2012 18:42:02 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> Message-ID: <5033BA5A.2090000@gfz-potsdam.de> On 21.08.2012 18:32, Matthew Knepley wrote: > > MUMPS takes only several minutes and 6 GB of memory to factorize it. > This factorization gives residual on the order of 10e-12 and > solution is indeed correct. > > Nevertheless, you're right, there is numerical null-space in this > matrix since it comes > from the discretization of equation that contains curl curl > operator, but practically this > case is not really the worst one. > > > This makes no sense whatsoever. How can you LU factor a matrix that > has a null space? Matt, I'm not sure that I correctly used term numerical null-space in my post. The equation is curl curl E + kE = -J, where k is a function of frequency and conductivity, whenever one of them becomes small this term gets vanishingly small thus we have problems since curl curl operator has nontrivial null-space by definition. So let's say solving this equation for low frequencies and for models containing air is difficult. What kind of magic is inside MUMPS I don't know, but it is able to handle such cases (e.g. SuperLU and PaStiX fail). Also, if it matters, I'm talking about LDLt factorization in MUMPS. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.richard.green at gmail.com Tue Aug 21 11:45:24 2012 From: kevin.richard.green at gmail.com (Kevin Green) Date: Tue, 21 Aug 2012 12:45:24 -0400 Subject: [petsc-users] Scaling a vector with an entry from a DMRedundant Message-ID: Hi all, I've created a Matshell multiplication that acts on a vector derived from a DMComposite which contains a DMDA and a DMRedundant of size 1 living on process 0. It does its job on a single process, but on multiple processes, the VecAXPY (see below) fails with the error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Scalar value must be same on all processes, argument # 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 9cac76b7c349ad267a1605750f02b8ef9d350ff1 HG Date: Tue Aug 21 10:07:43 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex3 on a arch-linu named lester by kg Tue Aug 21 12:36:10 2012 [0]PETSC ERROR: Libraries linked from /home/kg/lib/petsc-dev/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 11:54:03 2012 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-debug [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecAXPY() line 550 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: ArcContShellMult() line 1049 in ex3.c [0]PETSC ERROR: MatMult() line 2146 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 621 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPGMRESCycle() line 155 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: main() line 425 in ex3.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. Prodding the Matshell shows that a process that the DMRedundant doesn't "live" on has size 0. I guess the behaviour I expected is that every process would have space for the redundant entry, but all processes would be synchronized with the process where it lives upon access (or at least on a scatter, broadcast, ... something). The examples with DMRedundant are cases where only a single process needs to know the "redundant" part (so why is that called redundant?), and do not demonstrate this behaviour. Do I have to broadcast the information myself before I can scale a global vector? Is there something that might work better than DMRedundant? Any suggestions would be welcome. Cheers, Kevin Matshell multiplication routine: /* ArcContShellMult - MatMult overload for linear system of a pseudo-arclength continuation [du_post;dlam_post] = [dFdu dFdlam; du0 dlam0]*[du;dlam] Input Parameters: . A - The matshell . x - the state to operate on Output Parameters: . y - the result */ PetscErrorCode ArcContShellMult(Mat A,Vec x,Vec y) { PetscInt ierr; void *ctx; ArcContCtx *arc_ctx; Vec delta_U, delta_lam, delta_U_post, delta_lam_post; PetscScalar *del_lam_arr,lam_on_entry; PetscFunctionBegin; ierr = MatShellGetContext(A,&ctx); CHKERRQ(ierr); arc_ctx = (ArcContCtx*)(ctx); /* Get the components of the composite input and output */ ierr = DMCompositeGetAccess(arc_ctx->dmc,x,&delta_U,&delta_lam);CHKERRQ(ierr); ierr = DMCompositeGetAccess(arc_ctx->dmc,y,&delta_U_post,&delta_lam_post);CHKERRQ(ierr); /* Get lambda component as usable array */ ierr = VecGetArray(delta_lam,&del_lam_arr);CHKERRQ(ierr); lam_on_entry = del_lam_arr[0]; /* Multiply and add each of the components */ ierr = MatMult(arc_ctx->dFdu,delta_U,delta_U_post);CHKERRQ(ierr); ierr = VecAXPY(delta_U_post,del_lam_arr[0],arc_ctx->dFdlam);CHKERRQ(ierr); ierr = VecDot(arc_ctx->du0,delta_U,&del_lam_arr[0]);CHKERRQ(ierr); del_lam_arr[0] += arc_ctx->dlam0*lam_on_entry; /* Gather components back to the y vector */ ierr = VecRestoreArray(delta_lam,&del_lam_arr);CHKERRQ(ierr); ierr = VecCopy(delta_lam,delta_lam_post); ierr = DMCompositeRestoreAccess(arc_ctx->dmc,x,&delta_U,&delta_lam);CHKERRQ(ierr); ierr = DMCompositeRestoreAccess(arc_ctx->dmc,y,&delta_U_post,&delta_lam_post);CHKERRQ(ierr); PetscFunctionReturn(0); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Aug 21 11:47:49 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 21 Aug 2012 11:47:49 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <5033BA5A.2090000@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver wrote: > On 21.08.2012 18:32, Matthew Knepley wrote: > > MUMPS takes only several minutes and 6 GB of memory to factorize it. >> This factorization gives residual on the order of 10e-12 and solution is >> indeed correct. >> >> Nevertheless, you're right, there is numerical null-space in this matrix >> since it comes >> from the discretization of equation that contains curl curl operator, but >> practically this >> case is not really the worst one. >> > > This makes no sense whatsoever. How can you LU factor a matrix that has > a null space? > > > Matt, > > I'm not sure that I correctly used term numerical null-space in my post. > The equation is > > curl curl E + kE = -J, > > where k is a function of frequency and conductivity, whenever one of them > becomes small this term gets vanishingly small thus we have problems since > curl curl operator has nontrivial null-space by definition. So let's say > solving this equation for low frequencies and for models containing air is > difficult. > > What kind of magic is inside MUMPS I don't know, but it is able to handle > such cases (e.g. SuperLU and PaStiX fail). > > Also, if it matters, I'm talking about LDLt factorization in MUMPS. > > -- > Regards, > Alexander > > You can find Vasseur's talk on this exact subject here: http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 21 13:22:37 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Aug 2012 13:22:37 -0500 Subject: [petsc-users] Scaling a vector with an entry from a DMRedundant In-Reply-To: References: Message-ID: <62176177-4F30-4E85-9BF5-33F06632DD66@mcs.anl.gov> Send the source to ArcContShellMult to petsc-maint and we'll find you an alternative. Barry On Aug 21, 2012, at 11:45 AM, Kevin Green wrote: > Hi all, > > I've created a Matshell multiplication that acts on a vector derived from a DMComposite which contains a DMDA and a DMRedundant of size 1 living on process 0. It does its job on a single process, but on multiple processes, the VecAXPY (see below) fails with the error: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: 9cac76b7c349ad267a1605750f02b8ef9d350ff1 HG Date: Tue Aug 21 10:07:43 2012 -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex3 on a arch-linu named lester by kg Tue Aug 21 12:36:10 2012 > [0]PETSC ERROR: Libraries linked from /home/kg/lib/petsc-dev/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 11:54:03 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-debug > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: VecAXPY() line 550 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: ArcContShellMult() line 1049 in ex3.c > [0]PETSC ERROR: MatMult() line 2146 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 621 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPGMRESCycle() line 155 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 425 in ex3.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 1. > > Prodding the Matshell shows that a process that the DMRedundant doesn't "live" on has size 0. I guess the behaviour I expected is that every process would have space for the redundant entry, but all processes would be synchronized with the process where it lives upon access (or at least on a scatter, broadcast, ... something). > > The examples with DMRedundant are cases where only a single process needs to know the "redundant" part (so why is that called redundant?), and do not demonstrate this behaviour. > > Do I have to broadcast the information myself before I can scale a global vector? Is there something that might work better than DMRedundant? Any suggestions would be welcome. > > Cheers, > Kevin > > Matshell multiplication routine: > > /* > ArcContShellMult - MatMult overload for linear system of a pseudo-arclength continuation > [du_post;dlam_post] = [dFdu dFdlam; du0 dlam0]*[du;dlam] > > Input Parameters: > . A - The matshell > . x - the state to operate on > > Output Parameters: > . y - the result > */ > PetscErrorCode ArcContShellMult(Mat A,Vec x,Vec y) > { > PetscInt ierr; > void *ctx; > ArcContCtx *arc_ctx; > > Vec delta_U, delta_lam, delta_U_post, delta_lam_post; > PetscScalar *del_lam_arr,lam_on_entry; > PetscFunctionBegin; > ierr = MatShellGetContext(A,&ctx); CHKERRQ(ierr); > arc_ctx = (ArcContCtx*)(ctx); > > /* > Get the components of the composite input and output > */ > ierr = DMCompositeGetAccess(arc_ctx->dmc,x,&delta_U,&delta_lam);CHKERRQ(ierr); > ierr = DMCompositeGetAccess(arc_ctx->dmc,y,&delta_U_post,&delta_lam_post);CHKERRQ(ierr); > > /* > Get lambda component as usable array > */ > ierr = VecGetArray(delta_lam,&del_lam_arr);CHKERRQ(ierr); > lam_on_entry = del_lam_arr[0]; > > /* > Multiply and add each of the components > */ > ierr = MatMult(arc_ctx->dFdu,delta_U,delta_U_post);CHKERRQ(ierr); > ierr = VecAXPY(delta_U_post,del_lam_arr[0],arc_ctx->dFdlam);CHKERRQ(ierr); > > ierr = VecDot(arc_ctx->du0,delta_U,&del_lam_arr[0]);CHKERRQ(ierr); > del_lam_arr[0] += arc_ctx->dlam0*lam_on_entry; > > /* > Gather components back to the y vector > */ > ierr = VecRestoreArray(delta_lam,&del_lam_arr);CHKERRQ(ierr); > ierr = VecCopy(delta_lam,delta_lam_post); > ierr = DMCompositeRestoreAccess(arc_ctx->dmc,x,&delta_U,&delta_lam);CHKERRQ(ierr); > ierr = DMCompositeRestoreAccess(arc_ctx->dmc,y,&delta_U_post,&delta_lam_post);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > > From knepley at gmail.com Tue Aug 21 15:37:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Aug 2012 15:37:05 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 11:47 AM, Jack Poulson wrote: > On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> On 21.08.2012 18:32, Matthew Knepley wrote: >> >> MUMPS takes only several minutes and 6 GB of memory to factorize it. >>> This factorization gives residual on the order of 10e-12 and solution is >>> indeed correct. >>> >>> Nevertheless, you're right, there is numerical null-space in this matrix >>> since it comes >>> from the discretization of equation that contains curl curl operator, >>> but practically this >>> case is not really the worst one. >>> >> >> This makes no sense whatsoever. How can you LU factor a matrix that has >> a null space? >> >> >> Matt, >> >> I'm not sure that I correctly used term numerical null-space in my post. >> The equation is >> >> curl curl E + kE = -J, >> >> where k is a function of frequency and conductivity, whenever one of them >> becomes small this term gets vanishingly small thus we have problems since >> curl curl operator has nontrivial null-space by definition. So let's say >> solving this equation for low frequencies and for models containing air is >> difficult. >> >> What kind of magic is inside MUMPS I don't know, but it is able to handle >> such cases (e.g. SuperLU and PaStiX fail). >> >> Also, if it matters, I'm talking about LDLt factorization in MUMPS. >> >> -- >> Regards, >> Alexander >> >> You can find Vasseur's talk on this exact subject here: > http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf I was wrong, this is not nonsense. However, for curl curl the null space grows with matrix dimension, and as far as I can tell from the slides, the null space determination is not scalable (Jack correct me if I am wrong). Also, they gave no timings, so I suspect null space determination is slow. I don't think any other LU we have will do this, so if you have null spaces you are stuck with MUMPS. Matt > > Jack > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Aug 21 18:22:51 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 21 Aug 2012 18:22:51 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 3:37 PM, Matthew Knepley wrote: > On Tue, Aug 21, 2012 at 11:47 AM, Jack Poulson wrote: > >> On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver < >> agrayver at gfz-potsdam.de> wrote: >> >>> On 21.08.2012 18:32, Matthew Knepley wrote: >>> >>> MUMPS takes only several minutes and 6 GB of memory to factorize it. >>>> This factorization gives residual on the order of 10e-12 and solution >>>> is indeed correct. >>>> >>>> Nevertheless, you're right, there is numerical null-space in this >>>> matrix since it comes >>>> from the discretization of equation that contains curl curl operator, >>>> but practically this >>>> case is not really the worst one. >>>> >>> >>> This makes no sense whatsoever. How can you LU factor a matrix that >>> has a null space? >>> >>> >>> Matt, >>> >>> I'm not sure that I correctly used term numerical null-space in my post. >>> The equation is >>> >>> curl curl E + kE = -J, >>> >>> where k is a function of frequency and conductivity, whenever one of >>> them becomes small this term gets vanishingly small thus we have problems >>> since curl curl operator has nontrivial null-space by definition. So let's >>> say solving this equation for low frequencies and for models containing air >>> is difficult. >>> >>> What kind of magic is inside MUMPS I don't know, but it is able to >>> handle such cases (e.g. SuperLU and PaStiX fail). >>> >>> Also, if it matters, I'm talking about LDLt factorization in MUMPS. >>> >>> -- >>> Regards, >>> Alexander >>> >>> You can find Vasseur's talk on this exact subject here: >> http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf > > > I was wrong, this is not nonsense. However, for curl curl the null space > grows with matrix dimension, and > as far as I can tell from the slides, the null space determination is not > scalable (Jack correct me if I am wrong). > Also, they gave no timings, so I suspect null space determination is slow. > > I don't think any other LU we have will do this, so if you have null > spaces you are stuck with MUMPS. > > Matt > > It's not something that I've studied in detail, but I believe that it isn't that the behavior will not be that different from "difficult" nonsingular cases (i.e., where a large number of pivots do not satisfy the thresholding condition and must be delayed to the parent front). If the null space is large, then I would expect this to impact performance significantly. I would expect it to make load balancing much more difficult. In practice, this might lead to nonscalability, as it is sophisticated algorithm. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Aug 21 18:24:32 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 21 Aug 2012 18:24:32 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 6:22 PM, Jack Poulson wrote: > On Tue, Aug 21, 2012 at 3:37 PM, Matthew Knepley wrote: > >> On Tue, Aug 21, 2012 at 11:47 AM, Jack Poulson wrote: >> >>> On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver < >>> agrayver at gfz-potsdam.de> wrote: >>> >>>> On 21.08.2012 18:32, Matthew Knepley wrote: >>>> >>>> MUMPS takes only several minutes and 6 GB of memory to factorize it. >>>>> This factorization gives residual on the order of 10e-12 and solution >>>>> is indeed correct. >>>>> >>>>> Nevertheless, you're right, there is numerical null-space in this >>>>> matrix since it comes >>>>> from the discretization of equation that contains curl curl operator, >>>>> but practically this >>>>> case is not really the worst one. >>>>> >>>> >>>> This makes no sense whatsoever. How can you LU factor a matrix that >>>> has a null space? >>>> >>>> >>>> Matt, >>>> >>>> I'm not sure that I correctly used term numerical null-space in my post. >>>> The equation is >>>> >>>> curl curl E + kE = -J, >>>> >>>> where k is a function of frequency and conductivity, whenever one of >>>> them becomes small this term gets vanishingly small thus we have problems >>>> since curl curl operator has nontrivial null-space by definition. So let's >>>> say solving this equation for low frequencies and for models containing air >>>> is difficult. >>>> >>>> What kind of magic is inside MUMPS I don't know, but it is able to >>>> handle such cases (e.g. SuperLU and PaStiX fail). >>>> >>>> Also, if it matters, I'm talking about LDLt factorization in MUMPS. >>>> >>>> -- >>>> Regards, >>>> Alexander >>>> >>>> You can find Vasseur's talk on this exact subject here: >>> http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf >> >> >> I was wrong, this is not nonsense. However, for curl curl the null space >> grows with matrix dimension, and >> as far as I can tell from the slides, the null space determination is not >> scalable (Jack correct me if I am wrong). >> Also, they gave no timings, so I suspect null space determination is slow. >> >> I don't think any other LU we have will do this, so if you have null >> spaces you are stuck with MUMPS. >> >> Matt >> >> > > It's not something that I've studied in detail, but I believe that it > isn't that the behavior will not be that different from "difficult" > nonsingular cases (i.e., where a large number of pivots do not satisfy the > thresholding condition and must be delayed to the parent front). If the > null space is large, then I would expect this to impact performance > significantly. I would expect it to make load balancing much more > difficult. In practice, this might lead to nonscalability, as it is > sophisticated algorithm. > > Jack > Please ignore the atrocity that was my attempt at a first sentence in the previous email: the point is that the delayed pivot mechanism is also used within the standard threshold pivoted LU factorization. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Aug 21 18:46:16 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 21 Aug 2012 18:46:16 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: On Tue, Aug 21, 2012 at 6:24 PM, Jack Poulson wrote: > On Tue, Aug 21, 2012 at 6:22 PM, Jack Poulson wrote: > >> On Tue, Aug 21, 2012 at 3:37 PM, Matthew Knepley wrote: >> >>> On Tue, Aug 21, 2012 at 11:47 AM, Jack Poulson wrote: >>> >>>> On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver < >>>> agrayver at gfz-potsdam.de> wrote: >>>> >>>>> On 21.08.2012 18:32, Matthew Knepley wrote: >>>>> >>>>> MUMPS takes only several minutes and 6 GB of memory to factorize >>>>>> it. >>>>>> This factorization gives residual on the order of 10e-12 and solution >>>>>> is indeed correct. >>>>>> >>>>>> Nevertheless, you're right, there is numerical null-space in this >>>>>> matrix since it comes >>>>>> from the discretization of equation that contains curl curl operator, >>>>>> but practically this >>>>>> case is not really the worst one. >>>>>> >>>>> >>>>> This makes no sense whatsoever. How can you LU factor a matrix that >>>>> has a null space? >>>>> >>>>> >>>>> Matt, >>>>> >>>>> I'm not sure that I correctly used term numerical null-space in my >>>>> post. >>>>> The equation is >>>>> >>>>> curl curl E + kE = -J, >>>>> >>>>> where k is a function of frequency and conductivity, whenever one of >>>>> them becomes small this term gets vanishingly small thus we have problems >>>>> since curl curl operator has nontrivial null-space by definition. So let's >>>>> say solving this equation for low frequencies and for models containing air >>>>> is difficult. >>>>> >>>>> What kind of magic is inside MUMPS I don't know, but it is able to >>>>> handle such cases (e.g. SuperLU and PaStiX fail). >>>>> >>>>> Also, if it matters, I'm talking about LDLt factorization in MUMPS. >>>>> >>>>> -- >>>>> Regards, >>>>> Alexander >>>>> >>>>> You can find Vasseur's talk on this exact subject here: >>>> http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf >>> >>> >>> I was wrong, this is not nonsense. However, for curl curl the null space >>> grows with matrix dimension, and >>> as far as I can tell from the slides, the null space determination is >>> not scalable (Jack correct me if I am wrong). >>> Also, they gave no timings, so I suspect null space determination is >>> slow. >>> >>> I don't think any other LU we have will do this, so if you have null >>> spaces you are stuck with MUMPS. >>> >>> Matt >>> >>> >> >> It's not something that I've studied in detail, but I believe that it >> isn't that the behavior will not be that different from "difficult" >> nonsingular cases (i.e., where a large number of pivots do not satisfy the >> thresholding condition and must be delayed to the parent front). If the >> null space is large, then I would expect this to impact performance >> significantly. I would expect it to make load balancing much more >> difficult. In practice, this might lead to nonscalability, as it is >> sophisticated algorithm. >> >> Jack >> > > Please ignore the atrocity that was my attempt at a first sentence in the > previous email: the point is that the delayed pivot mechanism is also used > within the standard threshold pivoted LU factorization. > > Jack > Hmmm, I just noticed that slide 25 says that (at least in 2010) the root node is processed sequentially in order to handle nontrivial nullspaces. I would contact the developers, but the description of ICNTL(24) on pg. 29 of the most recent manual (http://mumps.enseeiht.fr/doc/userguide_4.10.0.pdf) makes it seem that the root node must still be factored sequentially if an exact zero pivot is detected. This will certainly limit the parallelism, as the computational complexity of factoring the root node is usually essentially the same as factoring the entire problem. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From zyzhang at nuaa.edu.cn Tue Aug 21 22:30:15 2012 From: zyzhang at nuaa.edu.cn (Zhang) Date: Wed, 22 Aug 2012 11:30:15 +0800 (CST) Subject: [petsc-users] Variable arguments for DMComposite functions Message-ID: <16f617b.7f0.1394c6167ae.Coremail.zyzhang@nuaa.edu.cn> Hi, I am using Petsc to write a multi-block CFD solver. Sometimes I just need access one or two specific DMs in a DMComposite, I just wonder how I can do this work other than to list all DMs in those DMComposite- functions with variable arguments, such as DMCompositeGetEntries(...), etc. Why the developer not choose the arguments by number of DM to be accessed and then an array of pointers to them? Or is it possible for user to rewrite our own version of the above functions, like DM* da_array[2]; DMCompositeGetEntries(user->pack,2,da_array); Thanks a lot for your attention, Zhenyu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 22 00:11:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 22 Aug 2012 00:11:41 -0500 Subject: [petsc-users] Variable arguments for DMComposite functions In-Reply-To: <16f617b.7f0.1394c6167ae.Coremail.zyzhang@nuaa.edu.cn> References: <16f617b.7f0.1394c6167ae.Coremail.zyzhang@nuaa.edu.cn> Message-ID: On Tue, Aug 21, 2012 at 10:30 PM, Zhang wrote: > > Hi, > > I am using Petsc to write a multi-block CFD solver. > Sometimes I just need access one or two specific DMs in a DMComposite, I > just wonder how I can do this work other than to list all DMs in those > DMComposite- functions with variable arguments, such as > > DMCompositeGetEntries(...), etc. > > Why the developer not choose the arguments by number of DM to be accessed and then an array of pointers to them? Or is it possible for user to rewrite our own version of the above functions, like > > DM* da_array[2]; > DMCompositeGetEntries(user->pack,2,da_array); > > There is also http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMCompositeGetEntriesArray.html You can write an application-specific accessor that pulls out only the ones you are interested in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Aug 22 02:40:49 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 22 Aug 2012 09:40:49 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> Message-ID: <50348D01.7060107@gfz-potsdam.de> On 21.08.2012 22:37, Matthew Knepley wrote: > I was wrong, this is not nonsense. However, for curl curl the null > space grows with matrix dimension, and > as far as I can tell from the slides, the null space determination is > not scalable (Jack correct me if I am wrong). > Also, they gave no timings, so I suspect null space determination is slow. > > I don't think any other LU we have will do this, so if you have null > spaces you are stuck with MUMPS. PARDISO can do this as well. -- Regards, Alexander From ccchiou2012 at gmail.com Wed Aug 22 02:52:55 2012 From: ccchiou2012 at gmail.com (C.-C. Chiou) Date: Wed, 22 Aug 2012 15:52:55 +0800 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: Dear Mathew, Thanks for your response! At line 645 in the source code, http://www.mcs.anl.gov/petsc/petsc-dev/include/petscsnes.h.html , some SNES routines for multi-block solver were listed. Is there any source code for specific testing PDE problem with these routines? In addition, is the KSP version available? Sincerely regards. Chuei-Ching Chiou On Wed, Aug 8, 2012 at 9:06 PM, Matthew Knepley wrote: > On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: > >> Dear all, >> >> For concerning more general geometry of physical domain, for some >> situations, >> we need to apply the multi-block mesh. The attached figure (or >> alternatively the following link) illustrates one of examples. >> >> https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg >> >> Have you ever dealt with such kind of case with application of the PETSc >> object 'DA' ? >> > > Our intention for this is that you use a DMComposite structure. Let us > know if the documentation is unclear > on anything, since it is somewhat new. > > Thanks, > > Matt > > >> Suppose we build multiple DA(or DM) for governing the multi-block data >> structure, >> in other words, we respectively build a DA for each block of mesh. >> Especially in parallel computing, >> how can we manage and control the communications between blocks of data >> accurately and efficiently? >> And, how can we ensure the correctness of parallel computational results? >> >> I wonder whether some possible application of PETSc object 'DA' (or DM) >> is sufficient for dealing with cases in multi-block mesh. >> >> >> Sincerely regards. >> >> >> Chuei-Ching Chiou >> >> >> -- >> Chuei-Ching Chiou, >> Ph. D student, >> Aerothermal & Plasma Physics Laboratory >> Department of Mechanical Engineering >> National Chiao Tung University >> Hsinchu 30010, TAIWAN >> TEL: +886-35712121 ext. 55175 >> E-mail: C CChiou2012 at gmail.com >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Chuei-Ching Chiou, Ph. D student, Aerothermal & Plasma Physics Laboratory Department of Mechanical Engineering National Chiao Tung University Hsinchu 30010, TAIWAN TEL: +886-35712121 ext. 55175 E-mail: C CChiou2012 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengshen186 at gmail.com Wed Aug 22 04:43:08 2012 From: fengshen186 at gmail.com (Feng Li) Date: Wed, 22 Aug 2012 17:43:08 +0800 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 Message-ID: The performance of petsc_3.3p2 is five times faster than petsc_dev-r24197. Why? It is tested with pflotran example problem 100x100x100. config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g -O2" --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT detail time: petsc_3.3p2 petsc_dev24197 walltime(sec) 28.116 110.25 Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Wed Aug 22 05:06:53 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 22 Aug 2012 11:06:53 +0100 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 In-Reply-To: References: Message-ID: Can you re-run both codes with: -log_summary please? A On Wed, Aug 22, 2012 at 10:43 AM, Feng Li wrote: > The performance of petsc_3.3p2 is five times faster than petsc_dev-r24197. > Why? > > It is tested with pflotran example problem 100x100x100. > config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g -O2" > --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT > > detail time: > petsc_3.3p2 petsc_dev24197 > walltime(sec) 28.116 110.25 > > Thank you! -- ------------------------------ This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. From knepley at gmail.com Wed Aug 22 05:50:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Aug 2012 05:50:57 -0500 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 2:52 AM, C.-C. Chiou wrote: > Dear Mathew, > > Thanks for your response! > > At line 645 in the source code, > http://www.mcs.anl.gov/petsc/petsc-dev/include/petscsnes.h.html , > some SNES routines for multi-block solver were listed. > Is there any source code for specific testing PDE problem with these > routines? > That is an experiment that does not yet work. > In addition, is the KSP version available? > Its called ASM. Matt > Sincerely regards. > > Chuei-Ching Chiou > > > On Wed, Aug 8, 2012 at 9:06 PM, Matthew Knepley wrote: > >> On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: >> >>> Dear all, >>> >>> For concerning more general geometry of physical domain, for some >>> situations, >>> we need to apply the multi-block mesh. The attached figure (or >>> alternatively the following link) illustrates one of examples. >>> >>> https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg >>> >>> Have you ever dealt with such kind of case with application of the PETSc >>> object 'DA' ? >>> >> >> Our intention for this is that you use a DMComposite structure. Let us >> know if the documentation is unclear >> on anything, since it is somewhat new. >> >> Thanks, >> >> Matt >> >> >>> Suppose we build multiple DA(or DM) for governing the multi-block data >>> structure, >>> in other words, we respectively build a DA for each block of mesh. >>> Especially in parallel computing, >>> how can we manage and control the communications between blocks of data >>> accurately and efficiently? >>> And, how can we ensure the correctness of parallel computational results? >>> >>> I wonder whether some possible application of PETSc object 'DA' (or DM) >>> is sufficient for dealing with cases in multi-block mesh. >>> >>> >>> Sincerely regards. >>> >>> >>> Chuei-Ching Chiou >>> >>> >>> -- >>> Chuei-Ching Chiou, >>> Ph. D student, >>> Aerothermal & Plasma Physics Laboratory >>> Department of Mechanical Engineering >>> National Chiao Tung University >>> Hsinchu 30010, TAIWAN >>> TEL: +886-35712121 ext. 55175 >>> E-mail: C CChiou2012 at gmail.com >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Chuei-Ching Chiou, > Ph. D student, > Aerothermal & Plasma Physics Laboratory > Department of Mechanical Engineering > National Chiao Tung University > Hsinchu 30010, TAIWAN > TEL: +886-35712121 ext. 55175 > E-mail: C CChiou2012 at gmail.com > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengshen186 at gmail.com Wed Aug 22 05:52:08 2012 From: fengshen186 at gmail.com (Feng Li) Date: Wed, 22 Aug 2012 18:52:08 +0800 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 In-Reply-To: References: Message-ID: Thank you for your help! The log_summary result is in the attachment. Please kindly check the attachment! 2012/8/22 Aron Ahmadia > Can you re-run both codes with: -log_summary please? > > A > > On Wed, Aug 22, 2012 at 10:43 AM, Feng Li wrote: > > The performance of petsc_3.3p2 is five times faster than > petsc_dev-r24197. > > Why? > > > > It is tested with pflotran example problem 100x100x100. > > config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g -O2" > > --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT > > > > detail time: > > petsc_3.3p2 petsc_dev24197 > > walltime(sec) 28.116 110.25 > > > > Thank you! > > -- > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_33p2 Type: application/octet-stream Size: 45416 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_dev Type: application/octet-stream Size: 45437 bytes Desc: not available URL: From knepley at gmail.com Wed Aug 22 06:07:33 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Aug 2012 06:07:33 -0500 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 In-Reply-To: References: Message-ID: On Wed, Aug 22, 2012 at 5:52 AM, Feng Li wrote: > Thank you for your help! > The log_summary result is in the attachment. > Please kindly check the attachment! > This is not a PETSc change, its a PFLOTRAN change. You have changed the way the Jacobian is evaluated. Look at the 3.3 log SNESSolve 18 1.0 1.2947e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 44 83 0 0 0 93100 0 0 0 94 SNESFunctionEval 39 1.0 3.3130e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 24 0 0 0 0 0 SNESJacobianEval 21 1.0 4.2020e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 30 0 0 0 0 0 SNESLineSearch 21 1.0 1.8140e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 13 0 0 0 0 3 KSPGMRESOrthog 470 1.0 8.2534e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 50 0 0 0 6 59 0 0 0 879 against the dev log SNESSolve 18 1.0 4.3973e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 40 83 0 0 0 98100 0 0 0 28 SNESFunctionEval 39 1.0 3.2787e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 7 0 0 0 0 0 SNESJacobianEval 21 1.0 3.5318e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 32 0 0 0 0 79 0 0 0 0 0 SNESLineSearch 21 1.0 1.8007e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 4 0 0 0 0 3 KSPGMRESOrthog 470 1.0 8.0665e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 50 0 0 0 2 59 0 0 0 899 The huge difference here comes from the Jacobian evaluation, which we do not control. Matt > 2012/8/22 Aron Ahmadia > >> Can you re-run both codes with: -log_summary please? >> >> A >> >> On Wed, Aug 22, 2012 at 10:43 AM, Feng Li wrote: >> > The performance of petsc_3.3p2 is five times faster than >> petsc_dev-r24197. >> > Why? >> > >> > It is tested with pflotran example problem 100x100x100. >> > config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g -O2" >> > --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT >> > >> > detail time: >> > petsc_3.3p2 petsc_dev24197 >> > walltime(sec) 28.116 110.25 >> > >> > Thank you! >> >> -- >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before >> printing >> this email. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 22 06:47:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 22 Aug 2012 06:47:38 -0500 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <50348D01.7060107@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> <50348D01.7060107@gfz-potsdam.de> Message-ID: On Wed, Aug 22, 2012 at 2:40 AM, Alexander Grayver wrote: > PARDISO can do this as well. You are welcome to write an interface. I'm not interested in doing that work for software with such screwy licensing. (Note that more people would probably use the MKL version, but it is not compatible with upstream.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 22 06:56:51 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 22 Aug 2012 06:56:51 -0500 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 In-Reply-To: References: Message-ID: Also, * ##########################################################* * # #* * # WARNING!!! #* * # #* * # This code was compiled with a debugging option, #* * # To get timing results run ./configure #* * # using --with-debugging=no, the performance will #* * # be generally two or three times faster. #* * # #* * ##########################################################* On Wed, Aug 22, 2012 at 6:07 AM, Matthew Knepley wrote: > On Wed, Aug 22, 2012 at 5:52 AM, Feng Li wrote: > >> Thank you for your help! >> The log_summary result is in the attachment. >> Please kindly check the attachment! >> > > This is not a PETSc change, its a PFLOTRAN change. You have changed the way > the Jacobian is evaluated. Look at the 3.3 log > > SNESSolve 18 1.0 1.2947e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 44 83 0 0 0 93100 0 0 0 94 > SNESFunctionEval 39 1.0 3.3130e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 11 0 0 0 0 24 0 0 0 0 0 > SNESJacobianEval 21 1.0 4.2020e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 14 0 0 0 0 30 0 0 0 0 0 > SNESLineSearch 21 1.0 1.8140e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 > 0.0e+00 6 0 0 0 0 13 0 0 0 0 3 > KSPGMRESOrthog 470 1.0 8.2534e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 3 50 0 0 0 6 59 0 0 0 879 > > against the dev log > > SNESSolve 18 1.0 4.3973e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 40 83 0 0 0 98100 0 0 0 28 > SNESFunctionEval 39 1.0 3.2787e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 3 0 0 0 0 7 0 0 0 0 0 > SNESJacobianEval 21 1.0 3.5318e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 32 0 0 0 0 79 0 0 0 0 0 > SNESLineSearch 21 1.0 1.8007e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 > 0.0e+00 2 0 0 0 0 4 0 0 0 0 3 > KSPGMRESOrthog 470 1.0 8.0665e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 1 50 0 0 0 2 59 0 0 0 899 > > The huge difference here comes from the Jacobian evaluation, which we do > not control. > > Matt > > >> 2012/8/22 Aron Ahmadia >> >>> Can you re-run both codes with: -log_summary please? >>> >>> A >>> >>> On Wed, Aug 22, 2012 at 10:43 AM, Feng Li wrote: >>> > The performance of petsc_3.3p2 is five times faster than >>> petsc_dev-r24197. >>> > Why? >>> > >>> > It is tested with pflotran example problem 100x100x100. >>> > config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g -O2" >>> > --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT >>> > >>> > detail time: >>> > petsc_3.3p2 petsc_dev24197 >>> > walltime(sec) 28.116 110.25 >>> > >>> > Thank you! >>> >>> -- >>> >>> ------------------------------ >>> This message and its contents, including attachments are intended solely >>> for the original recipient. If you are not the intended recipient or have >>> received this message in error, please notify me immediately and delete >>> this message from your computer system. Any unauthorized use or >>> distribution is prohibited. Please consider the environment before >>> printing >>> this email. >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Aug 22 07:28:44 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 22 Aug 2012 14:28:44 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> <50348D01.7060107@gfz-potsdam.de> Message-ID: <5034D07C.9050403@gfz-potsdam.de> On 22.08.2012 13:47, Jed Brown wrote: > On Wed, Aug 22, 2012 at 2:40 AM, Alexander Grayver > > wrote: > > PARDISO can do this as well. > > > You are welcome to write an interface. I'm not interested in doing > that work for software with such screwy licensing. > > (Note that more people would probably use the MKL version, but it is > not compatible with upstream.) Jed, This was just a piece of information for those who are interested and possibly dealing with similar problems. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Wed Aug 22 08:19:06 2012 From: renzhengyong at gmail.com (renzhengyong) Date: Wed, 22 Aug 2012 15:19:06 +0200 Subject: [petsc-users] MatMatSolve in sequential call In-Reply-To: <5034D07C.9050403@gfz-potsdam.de> References: <5028FE7E.30601@gfz-potsdam.de> <502917EC.7090600@gfz-potsdam.de> <50291D42.2020102@gfz-potsdam.de> <502A25C2.2070003@gfz-potsdam.de> <5033B680.1030109@gfz-potsdam.de> <5033BA5A.2090000@gfz-potsdam.de> <50348D01.7060107@gfz-potsdam.de> <5034D07C.9050403@gfz-potsdam.de> Message-ID: <5034DC4A.60805@gmail.com> ? 2012-8-22 14:28, Alexander Grayver ??: > On 22.08.2012 13:47, Jed Brown wrote: >> On Wed, Aug 22, 2012 at 2:40 AM, Alexander Grayver >> > wrote: >> >> PARDISO can do this as well. >> >> >> You are welcome to write an interface. I'm not interested in doing >> that work for software with such screwy licensing. >> >> (Note that more people would probably use the MKL version, but it is >> not compatible with upstream.) > Jed, > > This was just a piece of information for those who are interested and > possibly dealing with similar problems. > Hi, Alexander, Thanks for the info. It is as you said that the MKL parsido can do that job based on my numerical experiments. Bes wishes Zhengyong -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Thu Aug 23 05:25:20 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Thu, 23 Aug 2012 12:25:20 +0200 Subject: [petsc-users] Error on process numbers greater than 2 Message-ID: Hi, I am a new user and was running the example problems. The given example in the package, ex2 (src/ksp/ksp/examples/tutorials/ex2.c), works fine with two processors (mpiexec -n 2 ex2) but fails to work with any number greater than 2. For example: mpiexec -np 3 ex2, returns : [1]PETSC ERROR: PetscTrFreeDefault() called from PetscGatherMessageLengths() line 140 in src/sys/utils/mpimesg.c [1]PETSC ERROR: Block [id=0(80)] at address 0x7f93799a1a20 is corrupted (probably write past end of array) [1]PETSC ERROR: Block allocated in PetscGatherMessageLengths() line 115 in src/sys/utils/mpimesg.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Memory corruption! [1]PETSC ERROR: Corrupted memory! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs Thu Aug 23 12:21:58 2012 [1]PETSC ERROR: Libraries linked from /Users/bihs/petsc/arch-darwin-c-debug/lib [1]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 --download-f2cblaslapack [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: PetscTrFreeDefault() line 299 in src/sys/memory/mtr.c [1]PETSC ERROR: PetscGatherMessageLengths() line 140 in src/sys/utils/mpimesg.c [1]PETSC ERROR: VecScatterCreate_PtoS() line 1664 in src/vec/vec/utils/vpscat.c [1]PETSC ERROR: VecScatterCreate() line 1331 in src/vec/vec/utils/vscat.c [1]PETSC ERROR: MatSetUpMultiply_MPIAIJ() line 150 in src/mat/impls/aij/mpi/mmaij.c [1]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 680 in src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: MatAssemblyEnd() line 4866 in src/mat/interface/matrix.c [1]PETSC ERROR: main() line 107 in src/ksp/ksp/examples/tutorials/ex2.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] VecScatterCreate_PtoS line 1631 src/vec/vec/utils/vpscat.c [0]PETSC ERROR: [0] VecScatterCreate line 844 src/vec/vec/utils/vscat.c [0]PETSC ERROR: [0] MatSetUpMultiply_MPIAIJ line 26 src/mat/impls/aij/mpi/mmaij.c [0]PETSC ERROR: [0] MatAssemblyEnd_MPIAIJ line 646 src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] MatAssemblyEnd line 4857 src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs Thu Aug 23 12:21:58 2012 [0]PETSC ERROR: Libraries linked from /Users/bihs/petsc/arch-darwin-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 --download-f2cblaslapack [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] VecScatterCreate_PtoS line 1631 src/vec/vec/utils/vpscat.c [2]PETSC ERROR: [2] VecScatterCreate line 844 src/vec/vec/utils/vscat.c [2]PETSC ERROR: [2] MatSetUpMultiply_MPIAIJ line 26 src/mat/impls/aij/mpi/mmaij.c [2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 646 src/mat/impls/aij/mpi/mpiaij.c [2]PETSC ERROR: [2] MatAssemblyEnd line 4857 src/mat/interface/matrix.c [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Signal received! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs Thu Aug 23 12:21:58 2012 [2]PETSC ERROR: Libraries linked from /Users/bihs/petsc/arch-darwin-c-debug/lib [2]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 [2]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 --download-f2cblaslapack [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file My machine has two 6 core Intel Xeon processors and have been running other programs that use MPI to run 12 processes. Could someone shed some light on why this is happening? Thanks. -- Arun Kamath PhD Stipendiat NTNU, Trondheim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 23 06:34:59 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Aug 2012 06:34:59 -0500 Subject: [petsc-users] Error on process numbers greater than 2 In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 5:25 AM, Arun Kamath wrote: > Hi, > I am a new user and was running the example problems. > > The given example in the package, ex2 > (src/ksp/ksp/examples/tutorials/ex2.c), works fine with two processors > (mpiexec -n 2 ex2) but fails to work with any number greater than 2. For > example: > mpiexec -np 3 ex2, returns : > > [1]PETSC ERROR: PetscTrFreeDefault() called from > PetscGatherMessageLengths() line 140 in src/sys/utils/mpimesg.c > [1]PETSC ERROR: Block [id=0(80)] at address 0x7f93799a1a20 is corrupted > (probably write past end of array) > [1]PETSC ERROR: Block allocated in PetscGatherMessageLengths() line 115 in > src/sys/utils/mpimesg.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Memory corruption! > [1]PETSC ERROR: Corrupted memory! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs > Thu Aug 23 12:21:58 2012 > [1]PETSC ERROR: Libraries linked from > /Users/bihs/petsc/arch-darwin-c-debug/lib > [1]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 > [1]PETSC ERROR: Configure options > --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 > --download-f2cblaslapack > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: PetscTrFreeDefault() line 299 in src/sys/memory/mtr.c > [1]PETSC ERROR: PetscGatherMessageLengths() line 140 in > src/sys/utils/mpimesg.c > [1]PETSC ERROR: VecScatterCreate_PtoS() line 1664 in > src/vec/vec/utils/vpscat.c > [1]PETSC ERROR: VecScatterCreate() line 1331 in src/vec/vec/utils/vscat.c > [1]PETSC ERROR: MatSetUpMultiply_MPIAIJ() line 150 in > src/mat/impls/aij/mpi/mmaij.c > [1]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 680 in > src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: MatAssemblyEnd() line 4866 in src/mat/interface/matrix.c > [1]PETSC ERROR: main() line 107 in src/ksp/ksp/examples/tutorials/ex2.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 1. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecScatterCreate_PtoS line 1631 > src/vec/vec/utils/vpscat.c > [0]PETSC ERROR: [0] VecScatterCreate line 844 src/vec/vec/utils/vscat.c > [0]PETSC ERROR: [0] MatSetUpMultiply_MPIAIJ line 26 > src/mat/impls/aij/mpi/mmaij.c > [0]PETSC ERROR: [0] MatAssemblyEnd_MPIAIJ line 646 > src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: [0] MatAssemblyEnd line 4857 src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs > Thu Aug 23 12:21:58 2012 > [0]PETSC ERROR: Libraries linked from > /Users/bihs/petsc/arch-darwin-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 > --download-f2cblaslapack > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] VecScatterCreate_PtoS line 1631 > src/vec/vec/utils/vpscat.c > [2]PETSC ERROR: [2] VecScatterCreate line 844 src/vec/vec/utils/vscat.c > [2]PETSC ERROR: [2] MatSetUpMultiply_MPIAIJ line 26 > src/mat/impls/aij/mpi/mmaij.c > [2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 646 > src/mat/impls/aij/mpi/mpiaij.c > [2]PETSC ERROR: [2] MatAssemblyEnd line 4857 src/mat/interface/matrix.c > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Signal received! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs > Thu Aug 23 12:21:58 2012 > [2]PETSC ERROR: Libraries linked from > /Users/bihs/petsc/arch-darwin-c-debug/lib > [2]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012 > [2]PETSC ERROR: Configure options > --with-mpi-dir=/usr/local/include/openmpi --with-fc=0 > --download-f2cblaslapack > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > My machine has two 6 core Intel Xeon processors and have been running > other programs that use MPI to run 12 processes. > Could someone shed some light on why this is happening? > It looks like your MPI installation is bad. Try reconfiguring using --download-mpich Matt > Thanks. > > -- > Arun Kamath > PhD Stipendiat > NTNU, Trondheim > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Thu Aug 23 09:13:53 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Thu, 23 Aug 2012 16:13:53 +0200 Subject: [petsc-users] Error on process numbers greater than 2 Message-ID: That worked, thanks a lot. I had reconfigured OpenMPI to include fortran compilers and that wasn't done properly. From mirzadeh at gmail.com Thu Aug 23 13:25:49 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 23 Aug 2012 11:25:49 -0700 Subject: [petsc-users] Regarding changing vector/matrix type at runtime Message-ID: Hi guys, I've added a small function to only solve the linear system in parallel for problems that are not big enough to justify parallelizing the whole thing but would still take some time to solve in serial -- so far It's been handy. To prevent extra copy from my own format to petsc's, I'm using VecCreateMPIWithArray and MatCreateMPIWithSplitArray functions and they work with MPI on very few processes (like 4 or so). I have a couple of questions: 1) Is it worth considering using pThread and/or CUSP versions of the matrices for such problems? 2) If so, what would be a better approach. Using hardwired vector types in the code or using generic VecCreate functions to be able to change the type at run-time? 3) Are there equivalent versions of XXXWithArray functions for CUSP and pThread types? I don't seem to find them in manual page. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 23 13:40:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 23 Aug 2012 13:40:58 -0500 Subject: [petsc-users] Regarding changing vector/matrix type at runtime In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 1:25 PM, Mohammad Mirzadeh wrote: > Hi guys, > > I've added a small function to only solve the linear system in parallel > for problems that are not big enough to justify parallelizing the whole > thing but would still take some time to solve in serial -- so far It's been > handy. To prevent extra copy from my own format to petsc's, I'm using > VecCreateMPIWithArray and MatCreateMPIWithSplitArray functions and they > work with MPI on very few processes (like 4 or so). I have a couple of > questions: > > 1) Is it worth considering using pThread and/or CUSP versions of the > matrices for such problems? > The old pthread types are being removed in favor of threading support for normal formats. > 2) If so, what would be a better approach. Using hardwired vector types in > the code or using generic VecCreate functions to be able to change the type > at run-time? > Just use VecSetType()/MatSetType() to switch. You should really profile just using MatSetValues() to assemble. It's very likely that you are needlessly complicating your code by trying to skip a copy. > 3) Are there equivalent versions of XXXWithArray functions for CUSP and > pThread types? I don't seem to find them in manual page. > The CUSP case wouldn't help because chances are you are not holding memory in CUSP format living on the device. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Aug 23 13:51:57 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 23 Aug 2012 11:51:57 -0700 Subject: [petsc-users] Regarding changing vector/matrix type at runtime In-Reply-To: References: Message-ID: > The CUSP case wouldn't help because chances are you are not holding memory in CUSP format living on the device. That makes sense. Thanks for the help, Jed. On Thu, Aug 23, 2012 at 11:40 AM, Jed Brown wrote: > On Thu, Aug 23, 2012 at 1:25 PM, Mohammad Mirzadeh wrote: > >> Hi guys, >> >> I've added a small function to only solve the linear system in parallel >> for problems that are not big enough to justify parallelizing the whole >> thing but would still take some time to solve in serial -- so far It's been >> handy. To prevent extra copy from my own format to petsc's, I'm using >> VecCreateMPIWithArray and MatCreateMPIWithSplitArray functions and they >> work with MPI on very few processes (like 4 or so). I have a couple of >> questions: >> >> 1) Is it worth considering using pThread and/or CUSP versions of the >> matrices for such problems? >> > > The old pthread types are being removed in favor of threading support for > normal formats. > > >> 2) If so, what would be a better approach. Using hardwired vector types >> in the code or using generic VecCreate functions to be able to change the >> type at run-time? >> > > Just use VecSetType()/MatSetType() to switch. You should really profile > just using MatSetValues() to assemble. It's very likely that you are > needlessly complicating your code by trying to skip a copy. > > >> 3) Are there equivalent versions of XXXWithArray functions for CUSP and >> pThread types? I don't seem to find them in manual page. >> > > The CUSP case wouldn't help because chances are you are not holding memory > in CUSP format living on the device. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siaeleni at hotmail.com Thu Aug 23 17:30:17 2012 From: siaeleni at hotmail.com (Eleni Siampli) Date: Fri, 24 Aug 2012 01:30:17 +0300 Subject: [petsc-users] EPSGetConverge problem In-Reply-To: References: , , Message-ID: Hello, I am new to petsc and I am trying to calculate the eigenpairs. My code is like the following: ierr = EPSCreate(PETSC_COMM_WORLD, &eps); CHKERRQ(ierr); ierr = EPSSetOperators(eps, S, D); CHKERRQ(ierr); ierr = EPSSetProblemType(eps, EPS_GHEP); CHKERRQ(ierr); PetscInt nev = 300; ierr = EPSSetDimensions(eps, nev, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); ierr = EPSSetWhichEigenpairs(eps, EPS_SMALLEST_MAGNITUDE);CHKERRQ(ierr); ierr = EPSSolve(eps); CHKERRQ(ierr); ierr = EPSGetConverged(eps, &m_nconv);CHKERRQ(ierr); What I don't understand is why m_conv is greater than nev. when I try to print the m_conv out: PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %D\n\n",m_nconv); it gives me 399 instead of 300 (that I want). Thank you in advance, Eleni -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengshen186 at gmail.com Fri Aug 24 00:40:30 2012 From: fengshen186 at gmail.com (Feng Li) Date: Fri, 24 Aug 2012 13:40:30 +0800 Subject: [petsc-users] compare: performance of petsc_3.3p2 and petsc_dev-r24197 In-Reply-To: References: Message-ID: Thank you for your help! The problem is resolved smoothly according to your method. 2012/8/22 Jed Brown > Also, > > * ##########################################################* > * # #* > * # WARNING!!! #* > * # #* > * # This code was compiled with a debugging option, #* > * # To get timing results run ./configure #* > * # using --with-debugging=no, the performance will #* > * # be generally two or three times faster. #* > * # #* > * ##########################################################* > > > On Wed, Aug 22, 2012 at 6:07 AM, Matthew Knepley wrote: > >> On Wed, Aug 22, 2012 at 5:52 AM, Feng Li wrote: >> >>> Thank you for your help! >>> The log_summary result is in the attachment. >>> Please kindly check the attachment! >>> >> >> This is not a PETSc change, its a PFLOTRAN change. You have changed the >> way >> the Jacobian is evaluated. Look at the 3.3 log >> >> SNESSolve 18 1.0 1.2947e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 >> 0.0e+00 44 83 0 0 0 93100 0 0 0 94 >> SNESFunctionEval 39 1.0 3.3130e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 11 0 0 0 0 24 0 0 0 0 0 >> SNESJacobianEval 21 1.0 4.2020e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 14 0 0 0 0 30 0 0 0 0 0 >> SNESLineSearch 21 1.0 1.8140e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 >> 0.0e+00 6 0 0 0 0 13 0 0 0 0 3 >> KSPGMRESOrthog 470 1.0 8.2534e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 >> 0.0e+00 3 50 0 0 0 6 59 0 0 0 879 >> >> against the dev log >> >> SNESSolve 18 1.0 4.3973e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 >> 0.0e+00 40 83 0 0 0 98100 0 0 0 28 >> SNESFunctionEval 39 1.0 3.2787e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 3 0 0 0 0 7 0 0 0 0 0 >> SNESJacobianEval 21 1.0 3.5318e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 32 0 0 0 0 79 0 0 0 0 0 >> SNESLineSearch 21 1.0 1.8007e+00 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 >> 0.0e+00 2 0 0 0 0 4 0 0 0 0 3 >> KSPGMRESOrthog 470 1.0 8.0665e-01 1.0 7.25e+08 1.0 0.0e+00 0.0e+00 >> 0.0e+00 1 50 0 0 0 2 59 0 0 0 899 >> >> The huge difference here comes from the Jacobian evaluation, which we do >> not control. >> >> Matt >> >> >>> 2012/8/22 Aron Ahmadia >>> >>>> Can you re-run both codes with: -log_summary please? >>>> >>>> A >>>> >>>> On Wed, Aug 22, 2012 at 10:43 AM, Feng Li >>>> wrote: >>>> > The performance of petsc_3.3p2 is five times faster than >>>> petsc_dev-r24197. >>>> > Why? >>>> > >>>> > It is tested with pflotran example problem 100x100x100. >>>> > config parameter?--with-mpi=0 --CC=gcc --FC=gfortran --CCFLAGS="-g >>>> -O2" >>>> > --FCFLAGS="-g -O2" --blas-lapack-dir=$MKLROOT >>>> > >>>> > detail time: >>>> > petsc_3.3p2 petsc_dev24197 >>>> > walltime(sec) 28.116 110.25 >>>> > >>>> > Thank you! >>>> >>>> -- >>>> >>>> ------------------------------ >>>> This message and its contents, including attachments are intended solely >>>> for the original recipient. If you are not the intended recipient or >>>> have >>>> received this message in error, please notify me immediately and delete >>>> this message from your computer system. Any unauthorized use or >>>> distribution is prohibited. Please consider the environment before >>>> printing >>>> this email. >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- Thank you? Best regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengshen186 at gmail.com Fri Aug 24 00:43:30 2012 From: fengshen186 at gmail.com (Feng Li) Date: Fri, 24 Aug 2012 13:43:30 +0800 Subject: [petsc-users] Documents of petsc with GPU Message-ID: I search the related documents of petsc with GPU on the web. But I only get some incomplete documents. Can anyone give me some help of the document? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Aug 24 04:43:08 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 24 Aug 2012 11:43:08 +0200 Subject: [petsc-users] EPSGetConverge problem In-Reply-To: References: , , Message-ID: El 24/08/2012, a las 00:30, Eleni Siampli escribi?: > Hello, > > I am new to petsc and I am trying to calculate the eigenpairs. > My code is like the following: > > ierr = EPSCreate(PETSC_COMM_WORLD, &eps); CHKERRQ(ierr); > ierr = EPSSetOperators(eps, S, D); CHKERRQ(ierr); > ierr = EPSSetProblemType(eps, EPS_GHEP); CHKERRQ(ierr); > > > PetscInt nev = 300; > ierr = EPSSetDimensions(eps, nev, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); > ierr = EPSSetWhichEigenpairs(eps, EPS_SMALLEST_MAGNITUDE);CHKERRQ(ierr); > ierr = EPSSolve(eps); CHKERRQ(ierr); > > ierr = EPSGetConverged(eps, &m_nconv);CHKERRQ(ierr); > > What I don't understand is why m_conv is greater than nev. > when I try to print the m_conv out: > PetscPrintf(PETSC_COMM_WORLD," Number of converged eigenpairs: %D\n\n",m_nconv); > > it gives me 399 instead of 300 (that I want). > > > Thank you in advance, > > Eleni As stated in the users manual, the number of converged eigenpairs may be different from nev (otherwise we would not have a function EPSGetConverged). If nconv>nev then you can just take the first nev solutions. Are your problem matrices of order 399? Jose From knepley at gmail.com Fri Aug 24 07:04:35 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 24 Aug 2012 07:04:35 -0500 Subject: [petsc-users] Documents of petsc with GPU In-Reply-To: References: Message-ID: On Fri, Aug 24, 2012 at 12:43 AM, Feng Li wrote: > I search the related documents of petsc with GPU on the web. > But I only get some incomplete documents. > Can anyone give me some help of the document? > http://www.mcs.anl.gov/petsc/features/gpus.html Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Fri Aug 24 08:47:34 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Fri, 24 Aug 2012 15:47:34 +0200 Subject: [petsc-users] building Petsc in an IDE Message-ID: Hi! I wanted to know if it is possible to build a PETSc file from an IDE, like codelite or code blocks. On my attempt to do so after adding the libraries libpestc.a, libhypre.a, libf2cblas.a and libf2clapack.a to the link libraries and petsc/conf, petsc/include and pets/arch-darwin-c-debug/include to the complier search directories, the executable is not built and errors similar to the one below are returned. "Linking console executable: /./examples/tutorials/ex2 Undefined symbols for architecture x86_64: "_XCreateGC", referenced from: _PetscDrawXiSetGC in libpetsc.a(xinit.o) .......... ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status" Does anyone have a solution to this? Thanks. -- Arun Kamath NTNU, Trondheim -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 24 08:50:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 24 Aug 2012 08:50:22 -0500 Subject: [petsc-users] building Petsc in an IDE In-Reply-To: References: Message-ID: Configure found X11, but you aren't linking it. You have to link all the libraries returned by $ make getlinklibs The output will vary depending on your system and configuration. Here's one of mine -Wl,-rpath,/home/jed/petsc/mpich/lib -Wl,-rpath,/home/jed/petsc/mpich/lib -L/home/jed/petsc/mpich/lib -lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetscsys -Wl,-rpath,/home/jed/petsc/mpich/lib -L/home/jed/petsc/mpich/lib -ltriangle -lX11 -lchaco -lsuperlu_4.3 -lsuperlu_dist_3.0 -lparmetis -lmetis -Wl,-rpath,/home/jed/usr/elemental-mpich/lib -L/home/jed/usr/elemental-mpich/lib -lelemental -lplcg -lpmrrr -lamspub -lml -Wl,-rpath,/opt/mpich2/lib -L/opt/mpich2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -lmpichcxx -lstdc++ -lpthread -lspai -lspooles -lblacs -lHYPRE -lmpichcxx -lstdc++ -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -Wl,-rpath,/usr/lib -L/usr/lib -lcholmod -lamd -lcolamd -lcamd -lccolamd -lumfpack -lcholmod -lcamd -lccolamd -lcolamd -lamd -lsuitesparseconfig -llapack -lblas -lm -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lmpl -lpthread -lrt -lgcc_s -ldl On Fri, Aug 24, 2012 at 8:47 AM, Arun Kamath wrote: > Hi! > I wanted to know if it is possible to build a PETSc file from an IDE, like > codelite or code blocks. > On my attempt to do so after adding the libraries > libpestc.a, libhypre.a, libf2cblas.a and libf2clapack.a to the link > libraries and > petsc/conf, petsc/include and pets/arch-darwin-c-debug/include to the > complier search directories, the executable is not built and > errors similar to the one below are returned. > > "Linking console executable: /./examples/tutorials/ex2 > Undefined symbols for architecture x86_64: > "_XCreateGC", referenced from: > _PetscDrawXiSetGC in libpetsc.a(xinit.o) > .......... > ld: symbol(s) not found for architecture x86_64 > collect2: error: ld returned 1 exit status" > > Does anyone have a solution to this? > Thanks. > -- > Arun Kamath > NTNU, Trondheim > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 24 17:16:32 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 24 Aug 2012 15:16:32 -0700 Subject: [petsc-users] building Petsc in an IDE In-Reply-To: References: Message-ID: > $ make getlinklibs I suggest you define environment variables for your different builds and use them in your IDE's configuration setting. I do this with Qt Creator and I have been using my codes from two different linux machine and a also a Mac without any problem. Also remember that if you have several builds you should do make PETSC_DIR=/path/to/petsc PETSC_ARCH=-petsc-arch getlinklibs On Fri, Aug 24, 2012 at 6:50 AM, Jed Brown wrote: > Configure found X11, but you aren't linking it. You have to link all the > libraries returned by > > $ make getlinklibs > > The output will vary depending on your system and configuration. Here's > one of mine > > -Wl,-rpath,/home/jed/petsc/mpich/lib -Wl,-rpath,/home/jed/petsc/mpich/lib > -L/home/jed/petsc/mpich/lib -lpetscts -lpetscsnes -lpetscksp -lpetscdm > -lpetscmat -lpetscvec -lpetscsys -Wl,-rpath,/home/jed/petsc/mpich/lib > -L/home/jed/petsc/mpich/lib -ltriangle -lX11 -lchaco -lsuperlu_4.3 > -lsuperlu_dist_3.0 -lparmetis -lmetis > -Wl,-rpath,/home/jed/usr/elemental-mpich/lib > -L/home/jed/usr/elemental-mpich/lib -lelemental -lplcg -lpmrrr -lamspub > -lml -Wl,-rpath,/opt/mpich2/lib -L/opt/mpich2/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 > -L/usr/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -lmpichcxx -lstdc++ -lpthread > -lspai -lspooles -lblacs -lHYPRE -lmpichcxx -lstdc++ -lsundials_cvode > -lsundials_nvecserial -lsundials_nvecparallel -Wl,-rpath,/usr/lib > -L/usr/lib -lcholmod -lamd -lcolamd -lcamd -lccolamd -lumfpack -lcholmod > -lcamd -lccolamd -lcolamd -lamd -lsuitesparseconfig -llapack -lblas -lm > -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ > -ldl -lmpich -lopa -lmpl -lpthread -lrt -lgcc_s -ldl > > > On Fri, Aug 24, 2012 at 8:47 AM, Arun Kamath wrote: > >> Hi! >> I wanted to know if it is possible to build a PETSc file from an IDE, >> like codelite or code blocks. >> On my attempt to do so after adding the libraries >> libpestc.a, libhypre.a, libf2cblas.a and libf2clapack.a to the link >> libraries and >> petsc/conf, petsc/include and pets/arch-darwin-c-debug/include to the >> complier search directories, the executable is not built and >> errors similar to the one below are returned. >> >> "Linking console executable: /./examples/tutorials/ex2 >> Undefined symbols for architecture x86_64: >> "_XCreateGC", referenced from: >> _PetscDrawXiSetGC in libpetsc.a(xinit.o) >> .......... >> ld: symbol(s) not found for architecture x86_64 >> collect2: error: ld returned 1 exit status" >> >> Does anyone have a solution to this? >> Thanks. >> -- >> Arun Kamath >> NTNU, Trondheim >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Sun Aug 26 20:42:11 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Sun, 26 Aug 2012 21:42:11 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> Message-ID: <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Im getting segmentation faults when I call PetscSectionSetChart, gdb is telling me this: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 13 at address: 0x0000000000000000 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, line=296, function=0x1016bd470 "PetscSectionSetChart", file=0x1016bcbb8 "/Users/christina/Desktop/COLIN/Work/petsc-dev/src/vec/vec/impls/seq/vsection.c", dir=0x1016bca50 "") at /Users/christina/Desktop/COLIN/Work/petsc-dev/src/sys/memory/mtr.c:279 279 if (head->classid != CLASSID_VALUE) { (gdb) ba #0 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, line=296, function=0x1016bd470 "PetscSectionSetChart", file=0x1016bcbb8 "/Users/christina/Desktop/COLIN/Work/petsc-dev/src/vec/vec/impls/seq/vsection.c", dir=0x1016bca50 "") at /Users/christina/Desktop/COLIN/Work/petsc-dev/src/sys/memory/mtr.c:279 #1 0x000000010075120b in PetscSectionSetChart (s=0x102e06910, pStart=1, pEnd=0) at /Users/christina/Desktop/COLIN/Work/petsc-dev/src/vec/vec/impls/seq/vsection.c:296 #2 0x00000001006e99f3 in petscsectionsetchart_ (s=0x7fff5fbfee38, pStart=0x10165bc1c, pEnd=0x10165bc18, __ierr=0x7fff5fbfee4c) at /Users/christina/Desktop/COLIN/Work/petsc-dev/src/vec/vec/impls/seq/ftn-auto/vsectionf.c:148 #3 0x0000000100020c4c in setfs_ (fsind=, numfields=) at umacr8.F:96 Also, calling: call PetscSectionCreate(PETSC_COMM_WORLD,sec,ierr) call PetscSectionGetChart(sec,pStart,pEnd,ierr) returns pStart = 1 and pEnd = 0. Shouldn't PetscSectionCreate set pStart and pEnd to -1 and -1? I am a bit stumped here, any idea what is going on? thanks Colin Quoting Matthew Knepley : > On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe wrote: > >> I am just not seeing where in the section such an IS would go. As far as I >> can tell from the functions in vsection.c, the only item in the section >> which is an IS is bcIndices, defining points where bcs are applied. >> > > If you have no structure other than dof, you can make a Section where the > points > just number dofs. Then you just run through the ints p in your IS can call > > PetscSectionSetDof(section, p, 1) > > Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you > want, > and FS will use it. > > Matt > > >> 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); >> 39 (*s)->atlasLayout.comm = comm; >> 40 (*s)->atlasLayout.pStart = -1; >> 41 (*s)->atlasLayout.pEnd = -1; >> 42 (*s)->atlasLayout.numDof = 1; >> 43 (*s)->atlasDof = PETSC_NULL; >> 44 (*s)->atlasOff = PETSC_NULL; >> 45 (*s)->bc = PETSC_NULL; >> 46 (*s)->bcIndices = PETSC_NULL; >> 47 (*s)->setup = PETSC_FALSE; >> 48 (*s)->numFields = 0; >> 49 (*s)->fieldNames = PETSC_NULL; >> 50 (*s)->field = PETSC_NULL; >> >> Quoting Matthew Knepley : >> >> On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe >> >wrote: >>> >>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>> >>>>> everything sized >>>>> correctly, but handling topology and iteration for the residual >>>>> calculation >>>>> would be completely >>>>> up to you. >>>>> >>>>> Matt >>>>> >>>>> >>>> It would be really great to define the initial data layout using IS to >>>> define the equation numbers of the splits, while letting the new DM stuff >>>> handle creating all the nested splits given command line options. Is >>>> something like this possible? >>>> >>>> >>> This is just what I say above. Is something not working? >>> >>> Matt >>> >>> >>> Colin >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe >>> >>>>> >wrote: >>>>> >>>>> No, No, No. You do not have to write a DM implementation. >>>>> >>>>>> >>>>>> >>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>> it >>>>>>> to >>>>>>> the DM with DMSetDefaultSection. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>>> DMMesh? >>>>>> In other >>>>>> words is there a way to create the data layout for the physics without >>>>>> having to specify >>>>>> any information about the mesh? >>>>>> >>>>>> >>>>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>>> everything sized >>>>> correctly, but handling topology and iteration for the residual >>>>> calculation >>>>> would be completely >>>>> up to you. >>>>> >>>>> Matt >>>>> >>>>> >>>>> Thanks >>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev >>>>>> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>>> cjm2176 at columbia.edu >>>>>>>> >wrote: >>>>>>>> >>>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked format >>>>>>>> >>>>>>>> since >>>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or 4-10 >>>>>>>>> active >>>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>>> memory >>>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>>> problems >>>>>>>>> with >>>>>>>>> the unblocked format. >>>>>>>>> >>>>>>>>> Matt, sorry I am not getting this, but I am still not sure how the >>>>>>>>> DM >>>>>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>>>>> that >>>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>>> subsequent >>>>>>>>> calls >>>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>>> missing >>>>>>>>> is >>>>>>>>> how the first call to DMCreateFieldDecomposition is able to figure >>>>>>>>> out >>>>>>>>> which equations belong to which field just from command line info >>>>>>>>> such >>>>>>>>> as >>>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>>> defined? >>>>>>>>> >>>>>>>>> This might change slightly in the near future in petsc-dev to allow >>>>>>>>> >>>>>>>>> one to >>>>>>>> define splits using named fields. >>>>>>>> In any event, there has to be DM support to implement the >>>>>>>> decompositions >>>>>>>> over a particular mesh/problem over that mesh. >>>>>>>> With DMDA you can essentially get combinations of strided fields in >>>>>>>> each >>>>>>>> split. DMCOMPOSITE allows you >>>>>>>> to pull out combinations of the subproblems that were put in there to >>>>>>>> begin with. If you have your own mesh, you have to write >>>>>>>> a DM implementation around it to expose the available fields. >>>>>>>> >>>>>>>> >>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>> >>>>>>> >>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>> it >>>>>>> to >>>>>>> the DM with DMSetDefaultSection. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Dmitry. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>> Quoting Matthew Knepley : >>>>>>>>> >>>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee >>>>>>>> > >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Colin, >>>>>>>>>> >>>>>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>>>> >>>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>>> are >>>>>>>>>>> zero >>>>>>>>>>> modulo a value of unity on the diagonal and the restrained value >>>>>>>>>>> on >>>>>>>>>>> the >>>>>>>>>>> right-hand side). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -sg >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>>> >>>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>>> blocks. >>>>>>>>>>> >>>>>>>>>>>> The >>>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>>> petsc >>>>>>>>>>>> to >>>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>>> and >>>>>>>>>>>> time >>>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>>> has >>>>>>>>>>>> boundary >>>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>>> (unrestrained) >>>>>>>>>>>> equations. So then in general fields can't be extracted from the >>>>>>>>>>>> active >>>>>>>>>>>> equations using strided blocks and I am stuck with generating the >>>>>>>>>>>> index >>>>>>>>>>>> sets defining the splits on my own. Will it still be possible to >>>>>>>>>>>> make >>>>>>>>>>>> use >>>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>>> >>>>>>>>>>>> FEAP website: >>>>>>>>>>>> http://www.ce.berkeley.edu/**********projects/feap/ >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>>> knepley at gmail.com >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>>> >>>>>>>>>>>>> >wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now >>>>>>>>>>>>>> to >>>>>>>>>>>>>> create >>>>>>>>>>>>>> >>>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>>> >>>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> original >>>>>>>>>>>>>>> problem? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>>> define >>>>>>>>>>>>>> splits >>>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split all >>>>>>>>>>>>>> fields >>>>>>>>>>>>>> apart at once >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>>> any >>>>>>>>>>>>>> combination of fields, which from the command line is something >>>>>>>>>>>>>> like >>>>>>>>>>>>>> >>>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>>> >>>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>>> which >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>> which >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>>> Professor of Civil Engineering >>>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>>> >>>>>>>>>>> 779 Davis Hall >>>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>>> Department of Civil Engineering >>>>>>>>>>> University of California >>>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>>> >>>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >< >>>>>>>>>>> http://www.ce.berkeley.edu/~********sanjay >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>>> >>>>>>>>>>> >> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> ------------------------------**********----------------- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>> which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Colin McAuliffe >>>>>>>>> PhD Candidate >>>>>>>>> Columbia University >>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Colin McAuliffe >>>>>> PhD Candidate >>>>>> Columbia University >>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From ccchiou2012 at gmail.com Sun Aug 26 22:29:32 2012 From: ccchiou2012 at gmail.com (C.-C. Chiou) Date: Mon, 27 Aug 2012 11:29:32 +0800 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: Matt, Thank you. Now, I'm dealing with a work like that: (for example) ______ | | | | ________ | | | | | | f1 | f2 | |________ | | | | | | |_____ | (or please see the attached file "blocks_fig1.png" ) I have a physical domain partitioned into two blocks f1 and f2 (or more for other case), and I need to solve Navier-Stoke equations concerning on such domain by multi-block solver, with two connected structured meshes respectively for f1 and f2. I wonder whether case like the above for example is considered as one of experiment. Regards. Chuei-Ching Chiou On Wed, Aug 22, 2012 at 6:50 PM, Matthew Knepley wrote: > On Wed, Aug 22, 2012 at 2:52 AM, C.-C. Chiou wrote: > >> Dear Mathew, >> >> Thanks for your response! >> >> At line 645 in the source code, >> http://www.mcs.anl.gov/petsc/petsc-dev/include/petscsnes.h.html , >> some SNES routines for multi-block solver were listed. >> Is there any source code for specific testing PDE problem with these >> routines? >> > > That is an experiment that does not yet work. > > >> In addition, is the KSP version available? >> > > Its called ASM. > > Matt > > >> Sincerely regards. >> >> Chuei-Ching Chiou >> >> >> On Wed, Aug 8, 2012 at 9:06 PM, Matthew Knepley wrote: >> >>> On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: >>> >>>> Dear all, >>>> >>>> For concerning more general geometry of physical domain, for some >>>> situations, >>>> we need to apply the multi-block mesh. The attached figure (or >>>> alternatively the following link) illustrates one of examples. >>>> >>>> https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg >>>> >>>> Have you ever dealt with such kind of case with application of the >>>> PETSc object 'DA' ? >>>> >>> >>> Our intention for this is that you use a DMComposite structure. Let us >>> know if the documentation is unclear >>> on anything, since it is somewhat new. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Suppose we build multiple DA(or DM) for governing the multi-block data >>>> structure, >>>> in other words, we respectively build a DA for each block of mesh. >>>> Especially in parallel computing, >>>> how can we manage and control the communications between blocks of data >>>> accurately and efficiently? >>>> And, how can we ensure the correctness of parallel computational >>>> results? >>>> >>>> I wonder whether some possible application of PETSc object 'DA' (or DM) >>>> is sufficient for dealing with cases in multi-block mesh. >>>> >>>> >>>> Sincerely regards. >>>> >>>> >>>> Chuei-Ching Chiou >>>> >>>> >>>> -- >>>> Chuei-Ching Chiou, >>>> Ph. D student, >>>> Aerothermal & Plasma Physics Laboratory >>>> Department of Mechanical Engineering >>>> National Chiao Tung University >>>> Hsinchu 30010, TAIWAN >>>> TEL: +886-35712121 ext. 55175 >>>> E-mail: C CChiou2012 at gmail.com >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Chuei-Ching Chiou, >> Ph. D student, >> Aerothermal & Plasma Physics Laboratory >> Department of Mechanical Engineering >> National Chiao Tung University >> Hsinchu 30010, TAIWAN >> TEL: +886-35712121 ext. 55175 >> E-mail: C CChiou2012 at gmail.com >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Chuei-Ching Chiou, Ph. D student, Aerothermal & Plasma Physics Laboratory Department of Mechanical Engineering National Chiao Tung University Hsinchu 30010, TAIWAN TEL: +886-35712121 ext. 55175 E-mail: C CChiou2012 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blocks_fig1.png Type: image/png Size: 13628 bytes Desc: not available URL: From arun.kamath.m at gmail.com Mon Aug 27 09:53:30 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Mon, 27 Aug 2012 16:53:30 +0200 Subject: [petsc-users] building Petsc in an IDE Message-ID: <9BBC8109-DD7B-4B4A-AC1B-F89E466C02E1@gmail.com> That worked. Thanks a lot. From greatgs2008 at gmail.com Mon Aug 27 10:59:40 2012 From: greatgs2008 at gmail.com (shu guo) Date: Mon, 27 Aug 2012 11:59:40 -0400 Subject: [petsc-users] reduce a MPIBAIJ matrix to seqaij format Message-ID: Dear All, I am a new user of PetSc in fortran. I want to solve a ill conditioned matrix right now. And the matrix is now assembled as a MPIBAIJ matrix in a paralleled way. Now I want to use superlu (the serial direct solver) to solve it since its ill-conditioned characteristic. I tried MatConvert as call MatConvert (Kmatem, MATSEQAIJ, MAT_INITIAL_MATRIX, Kmatemseq) and it gives me a segmentation error. Can anyone give me some advice? BTW, I am using Petsc 2.3.3-p15 version, which do not have Schur preconditioner for ill-conditioned matrix. Also hope someone can give me some advice on how to solve this kind of matrix based on this petsc version. Thanks a lot! Shu From jedbrown at mcs.anl.gov Mon Aug 27 11:07:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 11:07:29 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Message-ID: On Sun, Aug 26, 2012 at 8:42 PM, Colin McAuliffe wrote: > Im getting segmentation faults when I call PetscSectionSetChart, gdb is > telling me this: > > Program received signal EXC_BAD_ACCESS, Could not access memory. > Reason: 13 at address: 0x0000000000000000 > 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, line=296, > function=0x1016bd470 "PetscSectionSetChart", > file=0x1016bcbb8 "/Users/christina/Desktop/** > COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", dir=0x1016bca50 > "") > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** > memory/mtr.c:279 > 279 if (head->classid != CLASSID_VALUE) { > (gdb) ba > #0 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, > line=296, function=0x1016bd470 "PetscSectionSetChart", > file=0x1016bcbb8 "/Users/christina/Desktop/** > COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", dir=0x1016bca50 > "") > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** > memory/mtr.c:279 > #1 0x000000010075120b in PetscSectionSetChart (s=0x102e06910, pStart=1, > pEnd=0) > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** > vec/impls/seq/vsection.c:296 > #2 0x00000001006e99f3 in petscsectionsetchart_ (s=0x7fff5fbfee38, > pStart=0x10165bc1c, pEnd=0x10165bc18, __ierr=0x7fff5fbfee4c) > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** > vec/impls/seq/ftn-auto/**vsectionf.c:148 > #3 0x0000000100020c4c in setfs_ (fsind= due to optimizations>, > numfields=) at > umacr8.F:96 > It would help to have a non-optimized trace. Have you run this under valgrind? > > > Also, calling: > > call PetscSectionCreate(PETSC_COMM_**WORLD,sec,ierr) > call PetscSectionGetChart(sec,**pStart,pEnd,ierr) > > returns pStart = 1 and pEnd = 0. Shouldn't PetscSectionCreate set pStart > and pEnd to -1 and -1? > > I am a bit stumped here, any idea what is going on? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Aug 27 11:34:18 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 27 Aug 2012 11:34:18 -0500 Subject: [petsc-users] reduce a MPIBAIJ matrix to seqaij format In-Reply-To: References: Message-ID: shu guo : > > I am a new user of PetSc in fortran. I want to solve a ill conditioned > matrix right now. And the matrix is now assembled as a MPIBAIJ matrix > in a paralleled way. Now I want to use superlu (the serial direct > solver) to solve it since its ill-conditioned characteristic. I tried > MatConvert as > > call MatConvert (Kmatem, MATSEQAIJ, MAT_INITIAL_MATRIX, > Kmatemseq) > and it gives me a segmentation error. Can anyone give me some advice? > call MatConvert (Kmatem, MATAIJ, MAT_INITIAL_MATRIX, Kmatemseq) or call MatConvert (Kmatem, MATMPIAIJ, MAT_INITIAL_MATRIX, Kmatemseq) > > BTW, I am using Petsc 2.3.3-p15 version, which do not have Schur > preconditioner for ill-conditioned matrix. Also hope someone can give > me some advice on how to solve this kind of matrix based on this petsc > version. Thanks a lot! > Petsc 2.3.3-p15 is too old. Please update to the latest petsc-3.3 and configure it with '--download-metis --download-parmetis --download-superlu --download-superlu_dist' Then use superlu_dist direct solver. Hong > > Shu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 11:34:22 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 12:34:22 -0400 Subject: [petsc-users] SNES Shell Problems Message-ID: Hi I am trying to use a PETSc snes shell object (with hopes of using it as the 'preconditioner' in athother snes ngmres object). I've upgraded to petsc3.3 but when I compile my code it complains that /home/mica/kenway/hg/SUmb/src/NKsolver/setupNKSolver2.F90:47: undefined reference to `snesshellsetsolve_' The code is fortran. I tried compiling ex35.c in the snes/examples directory and that went fine. I then looked at all the linking flags and split them out, one per line and compared it to the flags I'm using to compile my code. The two files are attached. (one.txt is my code and two is from 'make ex35'). A quick diff shows they are precisely identical. After a little more poking around, I found the snesshellf.c file in /src/snes/impls/shell/ftn-auto. From looking at that file, it appears the only function that actually got wrapped was: /* Definitions of Fortran Wrapper routines */ #if defined(__cplusplus) extern "C" { #endif void PETSC_STDCALL snesshellsetcontext_(SNES snes,void*ctx, int *__ierr ){ *__ierr = SNESShellSetContext( (SNES)PetscToPointer((snes) ),ctx); } #if defined(__cplusplus) } #endif This function, does in fact work. But all the other shell functions to not. Is is the snesshell functionality not implemented in fortran? What determines when bfort includes a file in the wrapper? Thanks, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- -Wl,-rpath,/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -L/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -lsuperlu_dist_3.0 -lparmetis -lmetis -llapack -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/ia32/lib -L/opt/intel/Compiler/11.1/072/ipp/ia32/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/ia32/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/072/tbb/ia32/cc4.1.0_libc2.4_kernel2.6.16.21/lib -Wl,-rpath,/usr/lib/gcc/i686-linux-gnu/4.4.5 -L/usr/lib/gcc/i686-linux-gnu/4.4.5 -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/ia32 -L/opt/intel/Compiler/11.1/072/lib/ia32 -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/32 -L/opt/intel/Compiler/11.1/072/mkl/lib/32 -Wl,-rpath,/usr/lib/i686-linux-gnu -L/usr/lib/i686-linux-gnu -lmpi_f90 -lmpi_f77 -lm -limf -lm -lifport -lifcoremt -lsvml -lm -lipgo -lirc -lirc_s -lm -lm -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -------------- next part -------------- -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -fno-inline -O0 -o ex35 ex35.o -Wl,-rpath,/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -L/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/nfs/mica/home/kenway/Downloads/petsc-3.3-p2/intel-real-debug/lib -lsuperlu_dist_3.0 -lparmetis -lmetis -llapack -lblas -Wl,-rpath,/usr/local/lib -L/usr/local/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/ia32/lib -L/opt/intel/Compiler/11.1/072/ipp/ia32/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/ia32/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/072/tbb/ia32/cc4.1.0_libc2.4_kernel2.6.16.21/lib -Wl,-rpath,/usr/lib/gcc/i686-linux-gnu/4.4.5 -L/usr/lib/gcc/i686-linux-gnu/4.4.5 -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/ia32 -L/opt/intel/Compiler/11.1/072/lib/ia32 -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/32 -L/opt/intel/Compiler/11.1/072/mkl/lib/32 -Wl,-rpath,/usr/lib/i686-linux-gnu -L/usr/lib/i686-linux-gnu -lmpi_f90 -lmpi_f77 -lm -limf -lm -lifport -lifcoremt -lsvml -lm -lipgo -lirc -lirc_s -lm -lm -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl From jedbrown at mcs.anl.gov Mon Aug 27 11:40:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 11:40:41 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 11:34 AM, Gaetan Kenway wrote: > Is is the snesshell functionality not implemented in fortran? It needs a custom Fortran binding which has not been written yet. Peter, can you add this to petsc-3.3? > What determines when bfort includes a file in the wrapper? When method comments start with /*@, bindings are automatically generated. When they start with /*@C, they are not generated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Mon Aug 27 11:40:57 2012 From: greatgs2008 at gmail.com (shu guo) Date: Mon, 27 Aug 2012 12:40:57 -0400 Subject: [petsc-users] reduce a MPIBAIJ matrix to seqaij format In-Reply-To: References: Message-ID: Hi Hong, Thanks for reply. I have a question on superlu_dist. Is this solver suitable for ill-conditioned matrix? I read some thread of past mailing list, Barry indicated that superlu is more adaptive with ill conditioned matrix. Thanks. Shu On Mon, Aug 27, 2012 at 12:34 PM, Hong Zhang wrote: > shu guo : >> >> >> I am a new user of PetSc in fortran. I want to solve a ill conditioned >> matrix right now. And the matrix is now assembled as a MPIBAIJ matrix >> in a paralleled way. Now I want to use superlu (the serial direct >> solver) to solve it since its ill-conditioned characteristic. I tried >> MatConvert as >> >> call MatConvert (Kmatem, MATSEQAIJ, MAT_INITIAL_MATRIX, >> Kmatemseq) >> and it gives me a segmentation error. Can anyone give me some advice? > > > call MatConvert (Kmatem, MATAIJ, MAT_INITIAL_MATRIX, Kmatemseq) > or > call MatConvert (Kmatem, MATMPIAIJ, MAT_INITIAL_MATRIX, Kmatemseq) >> >> >> BTW, I am using Petsc 2.3.3-p15 version, which do not have Schur >> preconditioner for ill-conditioned matrix. Also hope someone can give >> me some advice on how to solve this kind of matrix based on this petsc >> version. Thanks a lot! > > > Petsc 2.3.3-p15 is too old. Please update to the latest petsc-3.3 and > configure it > with '--download-metis --download-parmetis --download-superlu > --download-superlu_dist' > Then use superlu_dist direct solver. > > Hong >> >> >> Shu > > From jedbrown at mcs.anl.gov Mon Aug 27 11:43:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 11:43:09 -0500 Subject: [petsc-users] reduce a MPIBAIJ matrix to seqaij format In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 11:40 AM, shu guo wrote: > Thanks for reply. I have a question on superlu_dist. Is this solver > suitable for ill-conditioned matrix? > It depends how ill-conditioned > I read some thread of past > mailing list, Barry indicated that superlu is more adaptive with ill > conditioned matrix. Thanks. > SuperLU is a serial method, but it is typically more appropriate for very ill-conditioned systems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 12:09:49 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 13:09:49 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: That's what I had deduced. I tried modifying the source code taking out the 'C', but it still didn't seem to generate the required wrapper. Does the functions that get passed function pointer handle's require special handling? Gaetan On Mon, Aug 27, 2012 at 12:40 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 11:34 AM, Gaetan Kenway wrote: > >> Is is the snesshell functionality not implemented in fortran? > > > It needs a custom Fortran binding which has not been written yet. Peter, > can you add this to petsc-3.3? > > >> What determines when bfort includes a file in the wrapper? > > > When method comments start with /*@, bindings are automatically generated. > > When they start with /*@C, they are not generated. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 27 12:10:36 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 12:10:36 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway wrote: > That's what I had deduced. I tried modifying the source code taking out > the 'C', but it still didn't seem to generate the required wrapper. Does > the functions that get passed function pointer handle's require special > handling? It needs custom handling. Peter is working on it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 27 12:18:38 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Aug 2012 12:18:38 -0500 Subject: [petsc-users] Field split questions In-Reply-To: <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Message-ID: On Sun, Aug 26, 2012 at 8:42 PM, Colin McAuliffe wrote: > Im getting segmentation faults when I call PetscSectionSetChart, gdb is > telling me this: > > Program received signal EXC_BAD_ACCESS, Could not access memory. > Reason: 13 at address: 0x0000000000000000 > 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, line=296, > function=0x1016bd470 "PetscSectionSetChart", > file=0x1016bcbb8 "/Users/christina/Desktop/** > COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", dir=0x1016bca50 > "") > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** > memory/mtr.c:279 > 279 if (head->classid != CLASSID_VALUE) { > (gdb) ba > #0 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, > line=296, function=0x1016bd470 "PetscSectionSetChart", > file=0x1016bcbb8 "/Users/christina/Desktop/** > COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", dir=0x1016bca50 > "") > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** > memory/mtr.c:279 > #1 0x000000010075120b in PetscSectionSetChart (s=0x102e06910, pStart=1, > pEnd=0) > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** > vec/impls/seq/vsection.c:296 > #2 0x00000001006e99f3 in petscsectionsetchart_ (s=0x7fff5fbfee38, > pStart=0x10165bc1c, pEnd=0x10165bc18, __ierr=0x7fff5fbfee4c) > at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** > vec/impls/seq/ftn-auto/**vsectionf.c:148 > #3 0x0000000100020c4c in setfs_ (fsind= due to optimizations>, > numfields=) at > umacr8.F:96 > It looks like you never create the PetscSection in the call above. Can you send the code? > Also, calling: > > call PetscSectionCreate(PETSC_COMM_**WORLD,sec,ierr) > call PetscSectionGetChart(sec,**pStart,pEnd,ierr) > > returns pStart = 1 and pEnd = 0. Shouldn't PetscSectionCreate set pStart > and pEnd to -1 and -1? > Yes, that is true. I will create a Fortran example to check this. Matt > I am a bit stumped here, any idea what is going on? > > thanks > Colin > > Quoting Matthew Knepley : > > On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe > >wrote: >> >> I am just not seeing where in the section such an IS would go. As far as >>> I >>> can tell from the functions in vsection.c, the only item in the section >>> which is an IS is bcIndices, defining points where bcs are applied. >>> >>> >> If you have no structure other than dof, you can make a Section where the >> points >> just number dofs. Then you just run through the ints p in your IS can call >> >> PetscSectionSetDof(section, p, 1) >> >> Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you >> want, >> and FS will use it. >> >> Matt >> >> >> 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); >>> 39 (*s)->atlasLayout.comm = comm; >>> 40 (*s)->atlasLayout.pStart = -1; >>> 41 (*s)->atlasLayout.pEnd = -1; >>> 42 (*s)->atlasLayout.numDof = 1; >>> 43 (*s)->atlasDof = PETSC_NULL; >>> 44 (*s)->atlasOff = PETSC_NULL; >>> 45 (*s)->bc = PETSC_NULL; >>> 46 (*s)->bcIndices = PETSC_NULL; >>> 47 (*s)->setup = PETSC_FALSE; >>> 48 (*s)->numFields = 0; >>> 49 (*s)->fieldNames = PETSC_NULL; >>> 50 (*s)->field = PETSC_NULL; >>> >>> Quoting Matthew Knepley : >>> >>> On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe >> >>>> >wrote: >>>> >>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>> >>>>> >>>>> everything sized >>>>>> correctly, but handling topology and iteration for the residual >>>>>> calculation >>>>>> would be completely >>>>>> up to you. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> It would be really great to define the initial data layout using IS >>>>> to >>>>> define the equation numbers of the splits, while letting the new DM >>>>> stuff >>>>> handle creating all the nested splits given command line options. Is >>>>> something like this possible? >>>>> >>>>> >>>>> This is just what I say above. Is something not working? >>>> >>>> Matt >>>> >>>> >>>> Colin >>>> >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe < >>>>> cjm2176 at columbia.edu >>>>> >>>>> >wrote: >>>>>> >>>>>> No, No, No. You do not have to write a DM implementation. >>>>>> >>>>>> >>>>>>> >>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>>> it >>>>>>>> to >>>>>>>> the DM with DMSetDefaultSection. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>>>>> >>>>>>> DMMesh? >>>>>>> In other >>>>>>> words is there a way to create the data layout for the physics >>>>>>> without >>>>>>> having to specify >>>>>>> any information about the mesh? >>>>>>> >>>>>>> >>>>>>> Yes, just call DMSetDefaultSection(). However, note that you will >>>>>>> have >>>>>>> >>>>>> everything sized >>>>>> correctly, but handling topology and iteration for the residual >>>>>> calculation >>>>>> would be completely >>>>>> up to you. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Colin >>>>>>> >>>>>>> >>>>>>> >>>>>>> Quoting Matthew Knepley : >>>>>>> >>>>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev < >>>>>>> karpeev at mcs.anl.gov> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>>> >>>>>>>>> cjm2176 at columbia.edu >>>>>>>>> >wrote: >>>>>>>>> >>>>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked >>>>>>>>> format >>>>>>>>> >>>>>>>>> since >>>>>>>>> >>>>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or >>>>>>>>>> 4-10 >>>>>>>>>> active >>>>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>>>> memory >>>>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>>>> problems >>>>>>>>>> with >>>>>>>>>> the unblocked format. >>>>>>>>>> >>>>>>>>>> Matt, sorry I am not getting this, but I am still not sure how the >>>>>>>>>> DM >>>>>>>>>> interface works. I can see in the function PCFieldSplitSetDefaults >>>>>>>>>> that >>>>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>>>> subsequent >>>>>>>>>> calls >>>>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>>>> missing >>>>>>>>>> is >>>>>>>>>> how the first call to DMCreateFieldDecomposition is able to figure >>>>>>>>>> out >>>>>>>>>> which equations belong to which field just from command line info >>>>>>>>>> such >>>>>>>>>> as >>>>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>>>> defined? >>>>>>>>>> >>>>>>>>>> This might change slightly in the near future in petsc-dev to >>>>>>>>>> allow >>>>>>>>>> >>>>>>>>>> one to >>>>>>>>>> >>>>>>>>> define splits using named fields. >>>>>>>>> In any event, there has to be DM support to implement the >>>>>>>>> decompositions >>>>>>>>> over a particular mesh/problem over that mesh. >>>>>>>>> With DMDA you can essentially get combinations of strided fields in >>>>>>>>> each >>>>>>>>> split. DMCOMPOSITE allows you >>>>>>>>> to pull out combinations of the subproblems that were put in there >>>>>>>>> to >>>>>>>>> begin with. If you have your own mesh, you have to write >>>>>>>>> a DM implementation around it to expose the available fields. >>>>>>>>> >>>>>>>>> >>>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>>> >>>>>>>>> >>>>>>>> You just have to define the data layout in a PetscSection and attach >>>>>>>> it >>>>>>>> to >>>>>>>> the DM with DMSetDefaultSection. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Dmitry. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>> >>>>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee < >>>>>>>>>> s_g at berkeley.edu >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Colin, >>>>>>>>>>> >>>>>>>>>>> I you block the equations in FEAP, then the restrained BCs are >>>>>>>>>>> >>>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>>>> are >>>>>>>>>>>> zero >>>>>>>>>>>> modulo a value of unity on the diagonal and the restrained value >>>>>>>>>>>> on >>>>>>>>>>>> the >>>>>>>>>>>> right-hand side). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -sg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>>>> the >>>>>>>>>>>> >>>>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>>>> blocks. >>>>>>>>>>>> >>>>>>>>>>>> The >>>>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>>>> petsc >>>>>>>>>>>>> to >>>>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>>>> and >>>>>>>>>>>>> time >>>>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>>>> has >>>>>>>>>>>>> boundary >>>>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>>>> (unrestrained) >>>>>>>>>>>>> equations. So then in general fields can't be extracted from >>>>>>>>>>>>> the >>>>>>>>>>>>> active >>>>>>>>>>>>> equations using strided blocks and I am stuck with generating >>>>>>>>>>>>> the >>>>>>>>>>>>> index >>>>>>>>>>>>> sets defining the splits on my own. Will it still be possible >>>>>>>>>>>>> to >>>>>>>>>>>>> make >>>>>>>>>>>>> use >>>>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>>>> >>>>>>>>>>>>> FEAP website: >>>>>>>>>>>>> http://www.ce.berkeley.edu/************projects/feap/ >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Colin >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>>>> knepley at gmail.com >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure now >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> create >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> original >>>>>>>>>>>>>>>> problem? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> define >>>>>>>>>>>>>>> splits >>>>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>> fields >>>>>>>>>>>>>>> apart at once >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>>>> any >>>>>>>>>>>>>>> combination of fields, which from the command line is >>>>>>>>>>>>>>> something >>>>>>>>>>>>>>> like >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> experiments is infinitely more interesting than any results >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>>> which >>>>>>>>>>>>>> their >>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>>>> Professor of Civil Engineering >>>>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>>>> >>>>>>>>>>>> 779 Davis Hall >>>>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>>>> Department of Civil Engineering >>>>>>>>>>>> University of California >>>>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>>>> >>>>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>>> http://www.ce.berkeley.edu/~************sanjay >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>> http://berkeley.edu/~**sanjay**>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >< >>>>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>>> >> >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>> their >>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>> which >>>>>>>>>>> their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> Colin McAuliffe >>>>>>>>>> PhD Candidate >>>>>>>>>> Columbia University >>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 12:24:58 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 13:24:58 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Excellent. Thanks. The reason for using the snes shell solver is I would like to try out the ngmres snes. solver. The code I'm working with already has a has a non-linear multi-grid method, coded directly in the fortran source. My plan was to setup the the ngmres snes solver, then create a snes shell as the preconditioner that wraps the nonlienear multigrid solver already in place. Does this make sense? Is this the sort of usage you had in mind? Thanks, Gaetan On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway wrote: > >> That's what I had deduced. I tried modifying the source code taking out >> the 'C', but it still didn't seem to generate the required wrapper. Does >> the functions that get passed function pointer handle's require special >> handling? > > > It needs custom handling. Peter is working on it. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 27 13:19:19 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Aug 2012 13:19:19 -0500 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Message-ID: On Mon, Aug 27, 2012 at 12:18 PM, Matthew Knepley wrote: > On Sun, Aug 26, 2012 at 8:42 PM, Colin McAuliffe wrote: > >> Im getting segmentation faults when I call PetscSectionSetChart, gdb is >> telling me this: >> >> Program received signal EXC_BAD_ACCESS, Could not access memory. >> Reason: 13 at address: 0x0000000000000000 >> 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, >> line=296, function=0x1016bd470 "PetscSectionSetChart", >> file=0x1016bcbb8 "/Users/christina/Desktop/** >> COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", >> dir=0x1016bca50 "") >> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** >> memory/mtr.c:279 >> 279 if (head->classid != CLASSID_VALUE) { >> (gdb) ba >> #0 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, >> line=296, function=0x1016bd470 "PetscSectionSetChart", >> file=0x1016bcbb8 "/Users/christina/Desktop/** >> COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", >> dir=0x1016bca50 "") >> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** >> memory/mtr.c:279 >> #1 0x000000010075120b in PetscSectionSetChart (s=0x102e06910, pStart=1, >> pEnd=0) >> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** >> vec/impls/seq/vsection.c:296 >> #2 0x00000001006e99f3 in petscsectionsetchart_ (s=0x7fff5fbfee38, >> pStart=0x10165bc1c, pEnd=0x10165bc18, __ierr=0x7fff5fbfee4c) >> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** >> vec/impls/seq/ftn-auto/**vsectionf.c:148 >> #3 0x0000000100020c4c in setfs_ (fsind=> due to optimizations>, >> numfields=) at >> umacr8.F:96 >> > > It looks like you never create the PetscSection in the call above. Can you > send the code? > > >> Also, calling: >> >> call PetscSectionCreate(PETSC_COMM_**WORLD,sec,ierr) >> call PetscSectionGetChart(sec,**pStart,pEnd,ierr) >> >> returns pStart = 1 and pEnd = 0. Shouldn't PetscSectionCreate set pStart >> and pEnd to -1 and -1? >> > > Yes, that is true. I will create a Fortran example to check this. > Okay, there was a bug in the Fortran stub generation for this type. I have pushed a fix for this. In order to get these to rebuild, you need to rm $PETSC_DIR/$PETSC_ARCH/bin/bfort rm -rf $PETSC_DIR/externalpackages/sowing* cd $PETSC_DIR cd externalpackages hg clone http://petsc.cs.iit.edu/petsc/externalpackages/sowing-1.1.16/ ./$PETSC_ARCH/conf/reconfigure-$PETSC_ARCH.py Then you can run src/vec/vec/examples/tests/ex40f90.F Thanks Matt > Matt > > >> I am a bit stumped here, any idea what is going on? >> >> thanks >> Colin >> >> Quoting Matthew Knepley : >> >> On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe >> >wrote: >>> >>> I am just not seeing where in the section such an IS would go. As far >>>> as I >>>> can tell from the functions in vsection.c, the only item in the section >>>> which is an IS is bcIndices, defining points where bcs are applied. >>>> >>>> >>> If you have no structure other than dof, you can make a Section where the >>> points >>> just number dofs. Then you just run through the ints p in your IS can >>> call >>> >>> PetscSectionSetDof(section, p, 1) >>> >>> Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you >>> want, >>> and FS will use it. >>> >>> Matt >>> >>> >>> 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); >>>> 39 (*s)->atlasLayout.comm = comm; >>>> 40 (*s)->atlasLayout.pStart = -1; >>>> 41 (*s)->atlasLayout.pEnd = -1; >>>> 42 (*s)->atlasLayout.numDof = 1; >>>> 43 (*s)->atlasDof = PETSC_NULL; >>>> 44 (*s)->atlasOff = PETSC_NULL; >>>> 45 (*s)->bc = PETSC_NULL; >>>> 46 (*s)->bcIndices = PETSC_NULL; >>>> 47 (*s)->setup = PETSC_FALSE; >>>> 48 (*s)->numFields = 0; >>>> 49 (*s)->fieldNames = PETSC_NULL; >>>> 50 (*s)->field = PETSC_NULL; >>>> >>>> Quoting Matthew Knepley : >>>> >>>> On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe >>> >>>>> >wrote: >>>>> >>>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>>> >>>>>> >>>>>> everything sized >>>>>>> correctly, but handling topology and iteration for the residual >>>>>>> calculation >>>>>>> would be completely >>>>>>> up to you. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> It would be really great to define the initial data layout using IS >>>>>> to >>>>>> define the equation numbers of the splits, while letting the new DM >>>>>> stuff >>>>>> handle creating all the nested splits given command line options. Is >>>>>> something like this possible? >>>>>> >>>>>> >>>>>> This is just what I say above. Is something not working? >>>>> >>>>> Matt >>>>> >>>>> >>>>> Colin >>>>> >>>>>> >>>>>> Quoting Matthew Knepley : >>>>>> >>>>>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe < >>>>>> cjm2176 at columbia.edu >>>>>> >>>>>> >wrote: >>>>>>> >>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> You just have to define the data layout in a PetscSection and >>>>>>>>> attach >>>>>>>>> it >>>>>>>>> to >>>>>>>>> the DM with DMSetDefaultSection. >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>>>>>> >>>>>>>> DMMesh? >>>>>>>> In other >>>>>>>> words is there a way to create the data layout for the physics >>>>>>>> without >>>>>>>> having to specify >>>>>>>> any information about the mesh? >>>>>>>> >>>>>>>> >>>>>>>> Yes, just call DMSetDefaultSection(). However, note that you will >>>>>>>> have >>>>>>>> >>>>>>> everything sized >>>>>>> correctly, but handling topology and iteration for the residual >>>>>>> calculation >>>>>>> would be completely >>>>>>> up to you. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Colin >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Quoting Matthew Knepley : >>>>>>>> >>>>>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev < >>>>>>>> karpeev at mcs.anl.gov> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>>>> >>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>> >wrote: >>>>>>>>>> >>>>>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked >>>>>>>>>> format >>>>>>>>>> >>>>>>>>>> since >>>>>>>>>> >>>>>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or >>>>>>>>>>> 4-10 >>>>>>>>>>> active >>>>>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>>>>> memory >>>>>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>>>>> problems >>>>>>>>>>> with >>>>>>>>>>> the unblocked format. >>>>>>>>>>> >>>>>>>>>>> Matt, sorry I am not getting this, but I am still not sure how >>>>>>>>>>> the >>>>>>>>>>> DM >>>>>>>>>>> interface works. I can see in the function >>>>>>>>>>> PCFieldSplitSetDefaults >>>>>>>>>>> that >>>>>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>>>>> subsequent >>>>>>>>>>> calls >>>>>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>>>>> missing >>>>>>>>>>> is >>>>>>>>>>> how the first call to DMCreateFieldDecomposition is able to >>>>>>>>>>> figure >>>>>>>>>>> out >>>>>>>>>>> which equations belong to which field just from command line info >>>>>>>>>>> such >>>>>>>>>>> as >>>>>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>>>>> defined? >>>>>>>>>>> >>>>>>>>>>> This might change slightly in the near future in petsc-dev to >>>>>>>>>>> allow >>>>>>>>>>> >>>>>>>>>>> one to >>>>>>>>>>> >>>>>>>>>> define splits using named fields. >>>>>>>>>> In any event, there has to be DM support to implement the >>>>>>>>>> decompositions >>>>>>>>>> over a particular mesh/problem over that mesh. >>>>>>>>>> With DMDA you can essentially get combinations of strided fields >>>>>>>>>> in >>>>>>>>>> each >>>>>>>>>> split. DMCOMPOSITE allows you >>>>>>>>>> to pull out combinations of the subproblems that were put in >>>>>>>>>> there to >>>>>>>>>> begin with. If you have your own mesh, you have to write >>>>>>>>>> a DM implementation around it to expose the available fields. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> You just have to define the data layout in a PetscSection and >>>>>>>>> attach >>>>>>>>> it >>>>>>>>> to >>>>>>>>> the DM with DMSetDefaultSection. >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>> Dmitry. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Colin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee < >>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Colin, >>>>>>>>>>>> >>>>>>>>>>>> I you block the equations in FEAP, then the restrained BCs >>>>>>>>>>>> are >>>>>>>>>>>> >>>>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>>>>> are >>>>>>>>>>>>> zero >>>>>>>>>>>>> modulo a value of unity on the diagonal and the restrained >>>>>>>>>>>>> value >>>>>>>>>>>>> on >>>>>>>>>>>>> the >>>>>>>>>>>>> right-hand side). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -sg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>>>>> blocks. >>>>>>>>>>>>> >>>>>>>>>>>>> The >>>>>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>>>>> petsc >>>>>>>>>>>>>> to >>>>>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>>>>> and >>>>>>>>>>>>>> time >>>>>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>>>>> has >>>>>>>>>>>>>> boundary >>>>>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>>>>> (unrestrained) >>>>>>>>>>>>>> equations. So then in general fields can't be extracted from >>>>>>>>>>>>>> the >>>>>>>>>>>>>> active >>>>>>>>>>>>>> equations using strided blocks and I am stuck with generating >>>>>>>>>>>>>> the >>>>>>>>>>>>>> index >>>>>>>>>>>>>> sets defining the splits on my own. Will it still be possible >>>>>>>>>>>>>> to >>>>>>>>>>>>>> make >>>>>>>>>>>>>> use >>>>>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>>>>> >>>>>>>>>>>>>> FEAP website: >>>>>>>>>>>>>> http://www.ce.berkeley.edu/************projects/feap/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>>>>> knepley at gmail.com >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure >>>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> original >>>>>>>>>>>>>>>>> problem? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>> splits >>>>>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split >>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>> fields >>>>>>>>>>>>>>>> apart at once >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>>>>> any >>>>>>>>>>>>>>>> combination of fields, which from the command line is >>>>>>>>>>>>>>>> something >>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Colin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> experiments is infinitely more interesting than any results >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> experiments is infinitely more interesting than any results >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>>>>> Professor of Civil Engineering >>>>>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>>>>> >>>>>>>>>>>>> 779 Davis Hall >>>>>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>>>>> Department of Civil Engineering >>>>>>>>>>>>> University of California >>>>>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>>>>> >>>>>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>>>> http://www.ce.berkeley.edu/~************sanjay >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>> http://berkeley.edu/~**sanjay**>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >< >>>>>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >> >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their >>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>> which >>>>>>>>>>>> their >>>>>>>>>>>> experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>> Colin McAuliffe >>>>>>>>>>> PhD Candidate >>>>>>>>>>> Columbia University >>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>> which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> Colin McAuliffe >>>>>>>> PhD Candidate >>>>>>>> Columbia University >>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Colin McAuliffe >>>>>> PhD Candidate >>>>>> Columbia University >>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Colin McAuliffe >>>> PhD Candidate >>>> Columbia University >>>> Department of Civil Engineering and Engineering Mechanics >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> Colin McAuliffe >> PhD Candidate >> Columbia University >> Department of Civil Engineering and Engineering Mechanics >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Mon Aug 27 14:46:10 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Mon, 27 Aug 2012 13:46:10 -0600 Subject: [petsc-users] PETSc fortran type check Message-ID: Hello PETSC developers and users, I read on link http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html that approach 2 & 3 have some type check functionalities. My question is which subroutines are type checked and which are not. For example I guess that DMDACreate3d is checked (I'm using approach 3). Thanks and best wishes, Panruo Wu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Mon Aug 27 14:47:16 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Mon, 27 Aug 2012 13:47:16 -0600 Subject: [petsc-users] PETSc fortran type check In-Reply-To: References: Message-ID: Sorry a typo: last line: "checked" -> "not checked" On Mon, Aug 27, 2012 at 1:46 PM, Panruo Wu wrote: > Hello PETSC developers and users, > > I read on link > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html > > that approach 2 & 3 have some type check functionalities. My question is > which subroutines > are type checked and which are not. For example I guess that DMDACreate3d > is checked > (I'm using approach 3). > > Thanks and best wishes, > Panruo Wu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 27 14:47:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Aug 2012 14:47:49 -0500 Subject: [petsc-users] PETSc fortran type check In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 2:46 PM, Panruo Wu wrote: > Hello PETSC developers and users, > > I read on link > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html > > that approach 2 & 3 have some type check functionalities. My question is > which subroutines > are type checked and which are not. For example I guess that DMDACreate3d > is checked > (I'm using approach 3). > The ones with interface declarations. Matt > Thanks and best wishes, > Panruo Wu > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Mon Aug 27 14:59:46 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Mon, 27 Aug 2012 13:59:46 -0600 Subject: [petsc-users] PETSc fortran type check In-Reply-To: References: Message-ID: Mat, Could you give some details on how I can find all the unchecked(checked) subroutines so that I can narrow down my search area when segfaults occur? Thanks panruo On Mon, Aug 27, 2012 at 1:47 PM, Matthew Knepley wrote: > On Mon, Aug 27, 2012 at 2:46 PM, Panruo Wu wrote: > >> Hello PETSC developers and users, >> >> I read on link >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html >> >> that approach 2 & 3 have some type check functionalities. My question is >> which subroutines >> are type checked and which are not. For example I guess that >> DMDACreate3d is checked >> (I'm using approach 3). >> > > The ones with interface declarations. > > Matt > > >> Thanks and best wishes, >> Panruo Wu >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brune at mcs.anl.gov Mon Aug 27 14:59:54 2012 From: brune at mcs.anl.gov (Peter Brune) Date: Mon, 27 Aug 2012 14:59:54 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: I've tested and pushed a fix to petsc-3.3. If there are any problems, let me know. On Mon, Aug 27, 2012 at 12:24 PM, Gaetan Kenway wrote: > Excellent. Thanks. > > The reason for using the snes shell solver is I would like to try out the > ngmres snes. solver. The code I'm working with already has a has a > non-linear multi-grid method, coded directly in the fortran source. My plan > was to setup the the ngmres snes solver, then create a snes shell as the > preconditioner that wraps the nonlienear multigrid solver already in place. > > Does this make sense? Is this the sort of usage you had in mind? > > This is very much the usage we were intending for this. - Peter > Thanks, > > Gaetan > > > On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: > >> On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway > > wrote: >> >>> That's what I had deduced. I tried modifying the source code taking out >>> the 'C', but it still didn't seem to generate the required wrapper. Does >>> the functions that get passed function pointer handle's require special >>> handling? >> >> >> It needs custom handling. Peter is working on it. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 27 14:59:59 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Aug 2012 14:59:59 -0500 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: On Sun, Aug 26, 2012 at 10:29 PM, C.-C. Chiou wrote: > > Matt, > > Thank you. > > Now, I'm dealing with a work like that: (for example) > > ______ > | | > | | > ________ | | > | | | > | f1 | f2 | > |________ | | > | | > | | > |_____ | > > (or please see the attached file "blocks_fig1.png" ) > > I have a physical domain partitioned into two blocks f1 and f2 (or more > for other case), > and I need to solve Navier-Stoke equations concerning on such domain by > multi-block solver, > with two connected structured meshes respectively for f1 and f2. > > I wonder whether case like the above for example is considered as one of > experiment. > So for this case we have no easy support. You would have to manually construct a VecScatter object which encoded the coupling between the meshes, and there really are not any examples of this. Note that Hypre does have a specific interface for this (I think its SStruct). Thanks, Matt > Regards. > > > Chuei-Ching Chiou > > > > On Wed, Aug 22, 2012 at 6:50 PM, Matthew Knepley wrote: > >> On Wed, Aug 22, 2012 at 2:52 AM, C.-C. Chiou wrote: >> >>> Dear Mathew, >>> >>> Thanks for your response! >>> >>> At line 645 in the source code, >>> http://www.mcs.anl.gov/petsc/petsc-dev/include/petscsnes.h.html , >>> some SNES routines for multi-block solver were listed. >>> Is there any source code for specific testing PDE problem with these >>> routines? >>> >> >> That is an experiment that does not yet work. >> >> >>> In addition, is the KSP version available? >>> >> >> Its called ASM. >> >> Matt >> >> >>> Sincerely regards. >>> >>> Chuei-Ching Chiou >>> >>> >>> On Wed, Aug 8, 2012 at 9:06 PM, Matthew Knepley wrote: >>> >>>> On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: >>>> >>>>> Dear all, >>>>> >>>>> For concerning more general geometry of physical domain, for some >>>>> situations, >>>>> we need to apply the multi-block mesh. The attached figure (or >>>>> alternatively the following link) illustrates one of examples. >>>>> >>>>> https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg >>>>> >>>>> Have you ever dealt with such kind of case with application of the >>>>> PETSc object 'DA' ? >>>>> >>>> >>>> Our intention for this is that you use a DMComposite structure. Let us >>>> know if the documentation is unclear >>>> on anything, since it is somewhat new. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Suppose we build multiple DA(or DM) for governing the multi-block data >>>>> structure, >>>>> in other words, we respectively build a DA for each block of mesh. >>>>> Especially in parallel computing, >>>>> how can we manage and control the communications between blocks of >>>>> data accurately and efficiently? >>>>> And, how can we ensure the correctness of parallel computational >>>>> results? >>>>> >>>>> I wonder whether some possible application of PETSc object 'DA' (or >>>>> DM) is sufficient for dealing with cases in multi-block mesh. >>>>> >>>>> >>>>> Sincerely regards. >>>>> >>>>> >>>>> Chuei-Ching Chiou >>>>> >>>>> >>>>> -- >>>>> Chuei-Ching Chiou, >>>>> Ph. D student, >>>>> Aerothermal & Plasma Physics Laboratory >>>>> Department of Mechanical Engineering >>>>> National Chiao Tung University >>>>> Hsinchu 30010, TAIWAN >>>>> TEL: +886-35712121 ext. 55175 >>>>> E-mail: C CChiou2012 at gmail.com >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Chuei-Ching Chiou, >>> Ph. D student, >>> Aerothermal & Plasma Physics Laboratory >>> Department of Mechanical Engineering >>> National Chiao Tung University >>> Hsinchu 30010, TAIWAN >>> TEL: +886-35712121 ext. 55175 >>> E-mail: C CChiou2012 at gmail.com >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Chuei-Ching Chiou, > Ph. D student, > Aerothermal & Plasma Physics Laboratory > Department of Mechanical Engineering > National Chiao Tung University > Hsinchu 30010, TAIWAN > TEL: +886-35712121 ext. 55175 > E-mail: C CChiou2012 at gmail.com > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 27 15:00:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Aug 2012 15:00:44 -0500 Subject: [petsc-users] PETSc fortran type check In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 2:59 PM, Panruo Wu wrote: > Mat, > > Could you give some details on how I can find all the unchecked(checked) > subroutines so that I can narrow down my search area when > segfaults occur? > The right way to narrow this down is to use -start_in_debugger since you will immediately get the stack trace showing the offending method. Matt > Thanks > panruo > > On Mon, Aug 27, 2012 at 1:47 PM, Matthew Knepley wrote: > >> On Mon, Aug 27, 2012 at 2:46 PM, Panruo Wu wrote: >> >>> Hello PETSC developers and users, >>> >>> I read on link >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html >>> >>> that approach 2 & 3 have some type check functionalities. My question >>> is which subroutines >>> are type checked and which are not. For example I guess that >>> DMDACreate3d is checked >>> (I'm using approach 3). >>> >> >> The ones with interface declarations. >> >> Matt >> >> >>> Thanks and best wishes, >>> Panruo Wu >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Mon Aug 27 15:14:11 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Mon, 27 Aug 2012 14:14:11 -0600 Subject: [petsc-users] PETSc fortran type check In-Reply-To: References: Message-ID: Thanks Panruo On Mon, Aug 27, 2012 at 2:00 PM, Matthew Knepley wrote: > On Mon, Aug 27, 2012 at 2:59 PM, Panruo Wu wrote: > >> Mat, >> >> Could you give some details on how I can find all the unchecked(checked) >> subroutines so that I can narrow down my search area when >> segfaults occur? >> > > The right way to narrow this down is to use -start_in_debugger since you > will > immediately get the stack trace showing the offending method. > > Matt > > >> Thanks >> panruo >> >> On Mon, Aug 27, 2012 at 1:47 PM, Matthew Knepley wrote: >> >>> On Mon, Aug 27, 2012 at 2:46 PM, Panruo Wu wrote: >>> >>>> Hello PETSC developers and users, >>>> >>>> I read on link >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html >>>> >>>> that approach 2 & 3 have some type check functionalities. My question >>>> is which subroutines >>>> are type checked and which are not. For example I guess that >>>> DMDACreate3d is checked >>>> (I'm using approach 3). >>>> >>> >>> The ones with interface declarations. >>> >>> Matt >>> >>> >>>> Thanks and best wishes, >>>> Panruo Wu >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 16:06:53 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 17:06:53 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Thanks very much for the very quick response. I will pull the change from the mercurial repository and give it a try in the next couple of days. Gaetan On Mon, Aug 27, 2012 at 3:59 PM, Peter Brune wrote: > I've tested and pushed a fix to petsc-3.3. If there are any problems, let > me know. > > On Mon, Aug 27, 2012 at 12:24 PM, Gaetan Kenway wrote: > >> Excellent. Thanks. >> >> The reason for using the snes shell solver is I would like to try out the >> ngmres snes. solver. The code I'm working with already has a has a >> non-linear multi-grid method, coded directly in the fortran source. My plan >> was to setup the the ngmres snes solver, then create a snes shell as the >> preconditioner that wraps the nonlienear multigrid solver already in place. >> >> Does this make sense? Is this the sort of usage you had in mind? >> >> > This is very much the usage we were intending for this. > > - Peter > > >> Thanks, >> >> Gaetan >> >> >> On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: >> >>> On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway < >>> kenway at utias.utoronto.ca> wrote: >>> >>>> That's what I had deduced. I tried modifying the source code taking out >>>> the 'C', but it still didn't seem to generate the required wrapper. Does >>>> the functions that get passed function pointer handle's require special >>>> handling? >>> >>> >>> It needs custom handling. Peter is working on it. >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Aug 27 16:27:43 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 27 Aug 2012 16:27:43 -0500 (CDT) Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Message-ID: On Mon, 27 Aug 2012, Matthew Knepley wrote: > Okay, there was a bug in the Fortran stub generation for this type. I have > pushed a fix for this. In order to > get these to rebuild, you need to > hg clone http://petsc.cs.iit.edu/petsc/externalpackages/sowing-1.1.16/ Looks like this fix is needed for petsc-3.3. I created a new tarball for sowing and updated petsc-3.3 with it. Satish From kenway at utias.utoronto.ca Mon Aug 27 18:51:38 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 19:51:38 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Hi Again I've pulled the most recent 3.3 branch code, compiled and tried running with the shell solver. When I run just the snessolve() on the shell solver everything works just fine. However, when I try to combine it with NGMRES I get issues. Here's how I setup the solvers: ! ----------- Create Shell snes context ----------------- call SNESCreate(SUMB_COMM_WORLD, psnes, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the type to "shell" call SNESSetType(psnes, SNESSHELL, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the user-defined function for psnes to our funtion call SNESShellSetSolve(psnes, psnes_func, ierr) call EChk(ierr, __FILE__, __LINE__) ! ----------- Create main snes solver object ------------ call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the type to the nonlinear ngmres call SNESSetType(outer_snes, SNESNGMRES, ierr) call EChk(ierr, __FILE__, __LINE__) call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the above create psnes context to the 'pc' of the one just created call SNESSetPC(outer_snes, psnes, ierr) call EChk(ierr, __FILE__, __LINE__) call snessetfromoptions(outer_snes, ierr) call EChk(ierr, __FILE__, __LINE__) The actual commands I'm running are: call SNESSolve(outer_snes, rhs, wVec, ierr) ! this works when 'outer_snes' is replaced with 'psnes' call EChk(ierr,__FILE__,__LINE__) ! I'm pretty sure the function is correct since it runs through when just the shell solver is called. the "formFunction_snes" also seems to work ok by itself. The traceback I get is below: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] SNESSolve_Shell line 166 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c [0]PETSC ERROR: [0] SNESSolve line 3491 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NGMRES line 189 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c [0]PETSC ERROR: [0] SNESSolve line 3491 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c I can tell from the output it first calls the supplied 'formFunction' correctly and then has made its way to the SNESSolve_Shell command as expected. However, my guess is it actually dies at the following line: ierr = (*shell->solve)(snes,snes->vec_sol);CHKERRQ(ierr); I'm not quite sure what's going on. Something fishy with the vec_sol pointer or the solve command pointer? Do you have any examples of using NGMRES? Thanks, Gaetan On Mon, Aug 27, 2012 at 5:06 PM, Gaetan Kenway wrote: > Thanks very much for the very quick response. > > I will pull the change from the mercurial repository and give it a try in > the next couple of days. > > Gaetan > > > On Mon, Aug 27, 2012 at 3:59 PM, Peter Brune wrote: > >> I've tested and pushed a fix to petsc-3.3. If there are any problems, >> let me know. >> >> On Mon, Aug 27, 2012 at 12:24 PM, Gaetan Kenway > > wrote: >> >>> Excellent. Thanks. >>> >>> The reason for using the snes shell solver is I would like to try out >>> the ngmres snes. solver. The code I'm working with already has a has a >>> non-linear multi-grid method, coded directly in the fortran source. My plan >>> was to setup the the ngmres snes solver, then create a snes shell as the >>> preconditioner that wraps the nonlienear multigrid solver already in place. >>> >>> Does this make sense? Is this the sort of usage you had in mind? >>> >>> >> This is very much the usage we were intending for this. >> >> - Peter >> >> >>> Thanks, >>> >>> Gaetan >>> >>> >>> On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: >>> >>>> On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway < >>>> kenway at utias.utoronto.ca> wrote: >>>> >>>>> That's what I had deduced. I tried modifying the source code taking >>>>> out the 'C', but it still didn't seem to generate the required wrapper. >>>>> Does the functions that get passed function pointer handle's require >>>>> special handling? >>>> >>>> >>>> It needs custom handling. Peter is working on it. >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brune at mcs.anl.gov Mon Aug 27 19:00:11 2012 From: brune at mcs.anl.gov (Peter Brune) Date: Mon, 27 Aug 2012 19:00:11 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: I suspect that you could get past your problem by using SNESGetPC to create the initial psnes context from the outer solver context and then setting its type as you've done here. A more complete backtrace from the debugger might give me a better notion of exactly what is going wrong. You can get this output conveniently using the -on_error_attach_debugger. Thanks, - Peter On Mon, Aug 27, 2012 at 6:51 PM, Gaetan Kenway wrote: > Hi Again > > I've pulled the most recent 3.3 branch code, compiled and tried running > with the shell solver. When I run just the snessolve() on the shell solver > everything works just fine. However, when I try to combine it with NGMRES I > get issues. Here's how I setup the solvers: > > ! ----------- Create Shell snes context ----------------- > > > call SNESCreate(SUMB_COMM_WORLD, psnes, ierr) > call EChk(ierr, __FILE__, __LINE__) > > ! Set the type to "shell" > > > call SNESSetType(psnes, SNESSHELL, ierr) > call EChk(ierr, __FILE__, __LINE__) > > ! Set the user-defined function for psnes to our funtion > > > call SNESShellSetSolve(psnes, psnes_func, ierr) > call EChk(ierr, __FILE__, __LINE__) > > ! ----------- Create main snes solver object ------------ > > > call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) > call EChk(ierr, __FILE__, __LINE__) > > ! Set the type to the nonlinear ngmres > > > call SNESSetType(outer_snes, SNESNGMRES, ierr) > call EChk(ierr, __FILE__, __LINE__) > > call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) > call EChk(ierr, __FILE__, __LINE__) > > ! Set the above create psnes context to the 'pc' of the one just > created > > call SNESSetPC(outer_snes, psnes, ierr) > call EChk(ierr, __FILE__, __LINE__) > > call snessetfromoptions(outer_snes, ierr) > call EChk(ierr, __FILE__, __LINE__) > > > The actual commands I'm running are: > call SNESSolve(outer_snes, rhs, wVec, ierr) ! this works when > 'outer_snes' is replaced with 'psnes' > call EChk(ierr,__FILE__,__LINE__) ! > > > I'm pretty sure the function is correct since it runs through when just > the shell solver is called. the "formFunction_snes" also seems to work ok > by itself. The traceback I get is below: > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] SNESSolve_Shell line 166 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c > [0]PETSC ERROR: [0] SNESSolve line 3491 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NGMRES line 189 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c > [0]PETSC ERROR: [0] SNESSolve line 3491 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c > [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c > [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c > [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c > [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c > > I can tell from the output it first calls the supplied 'formFunction' > correctly and then has made its way to the SNESSolve_Shell command as > expected. However, my guess is it actually dies at the following line: > > ierr = (*shell->solve)(snes,snes->vec_sol);CHKERRQ(ierr); > > I'm not quite sure what's going on. Something fishy with the vec_sol > pointer or the solve command pointer? > > Do you have any examples of using NGMRES? > > Thanks, > > Gaetan > > On Mon, Aug 27, 2012 at 5:06 PM, Gaetan Kenway wrote: > >> Thanks very much for the very quick response. >> >> I will pull the change from the mercurial repository and give it a try in >> the next couple of days. >> >> Gaetan >> >> >> On Mon, Aug 27, 2012 at 3:59 PM, Peter Brune wrote: >> >>> I've tested and pushed a fix to petsc-3.3. If there are any problems, >>> let me know. >>> >>> On Mon, Aug 27, 2012 at 12:24 PM, Gaetan Kenway < >>> kenway at utias.utoronto.ca> wrote: >>> >>>> Excellent. Thanks. >>>> >>>> The reason for using the snes shell solver is I would like to try out >>>> the ngmres snes. solver. The code I'm working with already has a has a >>>> non-linear multi-grid method, coded directly in the fortran source. My plan >>>> was to setup the the ngmres snes solver, then create a snes shell as the >>>> preconditioner that wraps the nonlienear multigrid solver already in place. >>>> >>>> Does this make sense? Is this the sort of usage you had in mind? >>>> >>>> >>> This is very much the usage we were intending for this. >>> >>> - Peter >>> >>> >>>> Thanks, >>>> >>>> Gaetan >>>> >>>> >>>> On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: >>>> >>>>> On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway < >>>>> kenway at utias.utoronto.ca> wrote: >>>>> >>>>>> That's what I had deduced. I tried modifying the source code taking >>>>>> out the 'C', but it still didn't seem to generate the required wrapper. >>>>>> Does the functions that get passed function pointer handle's require >>>>>> special handling? >>>>> >>>>> >>>>> It needs custom handling. Peter is working on it. >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 19:31:46 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 20:31:46 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Hi Peter I changed by initialization to: ! ----------- Create main snes solver object ------------ call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the type to the nonlinear ngmres call SNESSetType(outer_snes, SNESNGMRES, ierr) call EChk(ierr, __FILE__, __LINE__) call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) call EChk(ierr, __FILE__, __LINE__) call snessetfromoptions(outer_snes, ierr) call EChk(ierr, __FILE__, __LINE__) ! ----------- Create Shell snes context ----------------- !call SNESCreate(SUMB_COMM_WORLD, psnes, ierr) !call EChk(ierr, __FILE__, __LINE__) ! Set the above create psnes context to the 'pc' of the one just created call SNESGetPC(outer_snes, psnes, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the type to "shell" call SNESSetType(psnes, SNESSHELL, ierr) call EChk(ierr, __FILE__, __LINE__) ! Set the user-defined function for psnes to our funtion call SNESShellSetSolve(psnes, psnes_func, ierr) call EChk(ierr, __FILE__, __LINE__) Unfortunately that didn't do anything to help. The output is still the same: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] SNESSolve_Shell line 166 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c [0]PETSC ERROR: [0] SNESSolve line 3491 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NGMRES line 189 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c [0]PETSC ERROR: [0] SNESSolve line 3491 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c Unfortunately I can run backtraces since the code is running from python and the -on_error_attach_debugger option has no effect when you're running from python. The petsc4py is initialized with: import petsc4py petsc4py.init(args=sys.argv) so other options are passed in just fine. I ran it with valgrind and got the following: ==15046== Jump to the invalid address stated on the next line ==15046== at 0x0: ??? ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in /tmp/tmpGQGZda/sumb.so) ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd which I'm not sure is of much additional use. Any thoughts? Gaetan On Mon, Aug 27, 2012 at 8:00 PM, Peter Brune wrote: > I suspect that you could get past your problem by using SNESGetPC to > create the initial psnes context from the outer solver context and then > setting its type as you've done here. > > A more complete backtrace from the debugger might give me a better notion > of exactly what is going wrong. You can get this output conveniently using > the -on_error_attach_debugger. > > Thanks, > > - Peter > > > On Mon, Aug 27, 2012 at 6:51 PM, Gaetan Kenway wrote: > >> Hi Again >> >> I've pulled the most recent 3.3 branch code, compiled and tried running >> with the shell solver. When I run just the snessolve() on the shell solver >> everything works just fine. However, when I try to combine it with NGMRES I >> get issues. Here's how I setup the solvers: >> >> ! ----------- Create Shell snes context ----------------- >> >> >> call SNESCreate(SUMB_COMM_WORLD, psnes, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> ! Set the type to "shell" >> >> >> call SNESSetType(psnes, SNESSHELL, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> ! Set the user-defined function for psnes to our funtion >> >> >> call SNESShellSetSolve(psnes, psnes_func, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> ! ----------- Create main snes solver object ------------ >> >> >> call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> ! Set the type to the nonlinear ngmres >> >> >> call SNESSetType(outer_snes, SNESNGMRES, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> ! Set the above create psnes context to the 'pc' of the one just >> created >> >> call SNESSetPC(outer_snes, psnes, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> call snessetfromoptions(outer_snes, ierr) >> call EChk(ierr, __FILE__, __LINE__) >> >> >> The actual commands I'm running are: >> call SNESSolve(outer_snes, rhs, wVec, ierr) ! this works when >> 'outer_snes' is replaced with 'psnes' >> call EChk(ierr,__FILE__,__LINE__) ! >> >> >> I'm pretty sure the function is correct since it runs through when just >> the shell solver is called. the "formFunction_snes" also seems to work ok >> by itself. The traceback I get is below: >> >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] SNESSolve_Shell line 166 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c >> [0]PETSC ERROR: [0] SNESSolve line 3491 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c >> [0]PETSC ERROR: [0] SNESSolve_NGMRES line 189 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c >> [0]PETSC ERROR: [0] SNESSolve line 3491 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c >> [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c >> [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c >> [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c >> [0]PETSC ERROR: [0] VecCreateMPIWithArray line 313 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/vec/vec/impls/mpi/pbvec.c >> >> I can tell from the output it first calls the supplied 'formFunction' >> correctly and then has made its way to the SNESSolve_Shell command as >> expected. However, my guess is it actually dies at the following line: >> >> ierr = (*shell->solve)(snes,snes->vec_sol);CHKERRQ(ierr); >> >> I'm not quite sure what's going on. Something fishy with the vec_sol >> pointer or the solve command pointer? >> >> Do you have any examples of using NGMRES? >> >> Thanks, >> >> Gaetan >> >> On Mon, Aug 27, 2012 at 5:06 PM, Gaetan Kenway wrote: >> >>> Thanks very much for the very quick response. >>> >>> I will pull the change from the mercurial repository and give it a try >>> in the next couple of days. >>> >>> Gaetan >>> >>> >>> On Mon, Aug 27, 2012 at 3:59 PM, Peter Brune wrote: >>> >>>> I've tested and pushed a fix to petsc-3.3. If there are any problems, >>>> let me know. >>>> >>>> On Mon, Aug 27, 2012 at 12:24 PM, Gaetan Kenway < >>>> kenway at utias.utoronto.ca> wrote: >>>> >>>>> Excellent. Thanks. >>>>> >>>>> The reason for using the snes shell solver is I would like to try out >>>>> the ngmres snes. solver. The code I'm working with already has a has a >>>>> non-linear multi-grid method, coded directly in the fortran source. My plan >>>>> was to setup the the ngmres snes solver, then create a snes shell as the >>>>> preconditioner that wraps the nonlienear multigrid solver already in place. >>>>> >>>>> Does this make sense? Is this the sort of usage you had in mind? >>>>> >>>>> >>>> This is very much the usage we were intending for this. >>>> >>>> - Peter >>>> >>>> >>>>> Thanks, >>>>> >>>>> Gaetan >>>>> >>>>> >>>>> On Mon, Aug 27, 2012 at 1:10 PM, Jed Brown wrote: >>>>> >>>>>> On Mon, Aug 27, 2012 at 12:09 PM, Gaetan Kenway < >>>>>> kenway at utias.utoronto.ca> wrote: >>>>>> >>>>>>> That's what I had deduced. I tried modifying the source code taking >>>>>>> out the 'C', but it still didn't seem to generate the required wrapper. >>>>>>> Does the functions that get passed function pointer handle's require >>>>>>> special handling? >>>>>> >>>>>> >>>>>> It needs custom handling. Peter is working on it. >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ccchiou2012 at gmail.com Mon Aug 27 20:13:06 2012 From: ccchiou2012 at gmail.com (C.-C. Chiou) Date: Tue, 28 Aug 2012 09:13:06 +0800 Subject: [petsc-users] The possibility of applying DA to governing multi-block data structure In-Reply-To: References: Message-ID: Matt, Thanks a lot! I'll try your suggestions. Chuei-Ching Chiou On Tue, Aug 28, 2012 at 3:59 AM, Matthew Knepley wrote: > On Sun, Aug 26, 2012 at 10:29 PM, C.-C. Chiou wrote: > >> >> Matt, >> >> Thank you. >> >> Now, I'm dealing with a work like that: (for example) >> >> ______ >> | | >> | | >> ________ | | >> | | | >> | f1 | f2 | >> |________ | | >> | | >> | | >> |_____ | >> >> (or please see the attached file "blocks_fig1.png" ) >> >> I have a physical domain partitioned into two blocks f1 and f2 (or more >> for other case), >> and I need to solve Navier-Stoke equations concerning on such domain by >> multi-block solver, >> with two connected structured meshes respectively for f1 and f2. >> >> I wonder whether case like the above for example is considered as one of >> experiment. >> > > So for this case we have no easy support. You would have to manually > construct a VecScatter > object which encoded the coupling between the meshes, and there really are > not any examples > of this. Note that Hypre does have a specific interface for this (I think > its SStruct). > > Thanks, > > Matt > > >> Regards. >> >> >> Chuei-Ching Chiou >> >> >> >> On Wed, Aug 22, 2012 at 6:50 PM, Matthew Knepley wrote: >> >>> On Wed, Aug 22, 2012 at 2:52 AM, C.-C. Chiou wrote: >>> >>>> Dear Mathew, >>>> >>>> Thanks for your response! >>>> >>>> At line 645 in the source code, >>>> http://www.mcs.anl.gov/petsc/petsc-dev/include/petscsnes.h.html , >>>> some SNES routines for multi-block solver were listed. >>>> Is there any source code for specific testing PDE problem with these >>>> routines? >>>> >>> >>> That is an experiment that does not yet work. >>> >>> >>>> In addition, is the KSP version available? >>>> >>> >>> Its called ASM. >>> >>> Matt >>> >>> >>>> Sincerely regards. >>>> >>>> Chuei-Ching Chiou >>>> >>>> >>>> On Wed, Aug 8, 2012 at 9:06 PM, Matthew Knepley wrote: >>>> >>>>> On Wed, Aug 8, 2012 at 2:30 AM, C.-C. Chiou wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> For concerning more general geometry of physical domain, for some >>>>>> situations, >>>>>> we need to apply the multi-block mesh. The attached figure (or >>>>>> alternatively the following link) illustrates one of examples. >>>>>> >>>>>> https://dl.dropbox.com/u/17189529/share_documents/IMAG0003.jpg >>>>>> >>>>>> Have you ever dealt with such kind of case with application of the >>>>>> PETSc object 'DA' ? >>>>>> >>>>> >>>>> Our intention for this is that you use a DMComposite structure. Let us >>>>> know if the documentation is unclear >>>>> on anything, since it is somewhat new. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Suppose we build multiple DA(or DM) for governing the multi-block >>>>>> data structure, >>>>>> in other words, we respectively build a DA for each block of mesh. >>>>>> Especially in parallel computing, >>>>>> how can we manage and control the communications between blocks of >>>>>> data accurately and efficiently? >>>>>> And, how can we ensure the correctness of parallel computational >>>>>> results? >>>>>> >>>>>> I wonder whether some possible application of PETSc object 'DA' (or >>>>>> DM) is sufficient for dealing with cases in multi-block mesh. >>>>>> >>>>>> >>>>>> Sincerely regards. >>>>>> >>>>>> >>>>>> Chuei-Ching Chiou >>>>>> >>>>>> >>>>>> -- >>>>>> Chuei-Ching Chiou, >>>>>> Ph. D student, >>>>>> Aerothermal & Plasma Physics Laboratory >>>>>> Department of Mechanical Engineering >>>>>> National Chiao Tung University >>>>>> Hsinchu 30010, TAIWAN >>>>>> TEL: +886-35712121 ext. 55175 >>>>>> E-mail: C CChiou2012 at gmail.com >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Chuei-Ching Chiou, >>>> Ph. D student, >>>> Aerothermal & Plasma Physics Laboratory >>>> Department of Mechanical Engineering >>>> National Chiao Tung University >>>> Hsinchu 30010, TAIWAN >>>> TEL: +886-35712121 ext. 55175 >>>> E-mail: C CChiou2012 at gmail.com >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Chuei-Ching Chiou, >> Ph. D student, >> Aerothermal & Plasma Physics Laboratory >> Department of Mechanical Engineering >> National Chiao Tung University >> Hsinchu 30010, TAIWAN >> TEL: +886-35712121 ext. 55175 >> E-mail: C CChiou2012 at gmail.com >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Chuei-Ching Chiou, Ph. D student, Aerothermal & Plasma Physics Laboratory Department of Mechanical Engineering National Chiao Tung University Hsinchu 30010, TAIWAN TEL: +886-35712121 ext. 55175 E-mail: C CChiou2012 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Mon Aug 27 21:32:21 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Mon, 27 Aug 2012 22:32:21 -0400 Subject: [petsc-users] Field split questions In-Reply-To: References: <20120808094809.3abj45yo0w4wwk8g@cubmail.cc.columbia.edu> <20120808205455.nnhhhs7peswkkk0s@cubmail.cc.columbia.edu> <6D31F841-8288-4A2F-89F8-EB2AD5C31B9D@mcs.anl.gov> <20120808215257.155ryj9jwwosww84@cubmail.cc.columbia.edu> <0FF3DFDA-A500-482F-A833-2A1E6CD9FF2C@mcs.anl.gov> <7CD9A81D-0B7E-4108-AC71-23F5EEF23FE9@mcs.anl.gov> <20120808232350.owflprhuf4wc40w0@cubmail.cc.columbia.edu> <20120809084143.a32quvywcogos40k@cubmail.cc.columbia.edu> <5023C752.1020401@berkeley.edu> <20120809110226.tqyerkkfzc4ssc00@cubmail.cc.columbia.edu> <20120813170637.184req1k84oc44w4@cubmail.cc.columbia.edu> <20120820081722.ki46vltwg0sk08ww@cubmail.cc.columbia.edu> <20120820084712.3pkldhfcgs4k4og0@cubmail.cc.columbia.edu> <20120826214211.s5nn1btb0g04k48w@cubmail.cc.columbia.edu> Message-ID: <20120827223221.13q5b5rku8ckokok@cubmail.cc.columbia.edu> Great! the example is working thanks. Colin Quoting Matthew Knepley : > On Mon, Aug 27, 2012 at 12:18 PM, Matthew Knepley wrote: > >> On Sun, Aug 26, 2012 at 8:42 PM, Colin McAuliffe >> wrote: >> >>> Im getting segmentation faults when I call PetscSectionSetChart, gdb is >>> telling me this: >>> >>> Program received signal EXC_BAD_ACCESS, Could not access memory. >>> Reason: 13 at address: 0x0000000000000000 >>> 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, >>> line=296, function=0x1016bd470 "PetscSectionSetChart", >>> file=0x1016bcbb8 "/Users/christina/Desktop/** >>> COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", >>> dir=0x1016bca50 "") >>> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** >>> memory/mtr.c:279 >>> 279 if (head->classid != CLASSID_VALUE) { >>> (gdb) ba >>> #0 0x00000001005f9585 in PetscTrFreeDefault (aa=0x4008000000000000, >>> line=296, function=0x1016bd470 "PetscSectionSetChart", >>> file=0x1016bcbb8 "/Users/christina/Desktop/** >>> COLIN/Work/petsc-dev/src/vec/**vec/impls/seq/vsection.c", >>> dir=0x1016bca50 "") >>> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/sys/** >>> memory/mtr.c:279 >>> #1 0x000000010075120b in PetscSectionSetChart (s=0x102e06910, pStart=1, >>> pEnd=0) >>> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** >>> vec/impls/seq/vsection.c:296 >>> #2 0x00000001006e99f3 in petscsectionsetchart_ (s=0x7fff5fbfee38, >>> pStart=0x10165bc1c, pEnd=0x10165bc18, __ierr=0x7fff5fbfee4c) >>> at /Users/christina/Desktop/**COLIN/Work/petsc-dev/src/vec/** >>> vec/impls/seq/ftn-auto/**vsectionf.c:148 >>> #3 0x0000000100020c4c in setfs_ (fsind=>> due to optimizations>, >>> numfields=) at >>> umacr8.F:96 >>> >> >> It looks like you never create the PetscSection in the call above. Can you >> send the code? >> >> >>> Also, calling: >>> >>> call PetscSectionCreate(PETSC_COMM_**WORLD,sec,ierr) >>> call PetscSectionGetChart(sec,**pStart,pEnd,ierr) >>> >>> returns pStart = 1 and pEnd = 0. Shouldn't PetscSectionCreate set pStart >>> and pEnd to -1 and -1? >>> >> >> Yes, that is true. I will create a Fortran example to check this. >> > > Okay, there was a bug in the Fortran stub generation for this type. I have > pushed a fix for this. In order to > get these to rebuild, you need to > > rm $PETSC_DIR/$PETSC_ARCH/bin/bfort > rm -rf $PETSC_DIR/externalpackages/sowing* > cd $PETSC_DIR > cd externalpackages > hg clone http://petsc.cs.iit.edu/petsc/externalpackages/sowing-1.1.16/ > ./$PETSC_ARCH/conf/reconfigure-$PETSC_ARCH.py > > Then you can run src/vec/vec/examples/tests/ex40f90.F > > Thanks > > Matt > > >> Matt >> >> >>> I am a bit stumped here, any idea what is going on? >>> >>> thanks >>> Colin >>> >>> Quoting Matthew Knepley : >>> >>> On Mon, Aug 20, 2012 at 7:47 AM, Colin McAuliffe >>> >wrote: >>>> >>>> I am just not seeing where in the section such an IS would go. As far >>>>> as I >>>>> can tell from the functions in vsection.c, the only item in the section >>>>> which is an IS is bcIndices, defining points where bcs are applied. >>>>> >>>>> >>>> If you have no structure other than dof, you can make a Section where the >>>> points >>>> just number dofs. Then you just run through the ints p in your IS can >>>> call >>>> >>>> PetscSectionSetDof(section, p, 1) >>>> >>>> Then PetscSectionSetUp(). Stick this in a DMSHELL, or really any DM you >>>> want, >>>> and FS will use it. >>>> >>>> Matt >>>> >>>> >>>> 38 ierr = PetscNew(struct _n_PetscSection, s);CHKERRQ(ierr); >>>>> 39 (*s)->atlasLayout.comm = comm; >>>>> 40 (*s)->atlasLayout.pStart = -1; >>>>> 41 (*s)->atlasLayout.pEnd = -1; >>>>> 42 (*s)->atlasLayout.numDof = 1; >>>>> 43 (*s)->atlasDof = PETSC_NULL; >>>>> 44 (*s)->atlasOff = PETSC_NULL; >>>>> 45 (*s)->bc = PETSC_NULL; >>>>> 46 (*s)->bcIndices = PETSC_NULL; >>>>> 47 (*s)->setup = PETSC_FALSE; >>>>> 48 (*s)->numFields = 0; >>>>> 49 (*s)->fieldNames = PETSC_NULL; >>>>> 50 (*s)->field = PETSC_NULL; >>>>> >>>>> Quoting Matthew Knepley : >>>>> >>>>> On Mon, Aug 20, 2012 at 7:17 AM, Colin McAuliffe >>>> >>>>>> >wrote: >>>>>> >>>>>> Yes, just call DMSetDefaultSection(). However, note that you will have >>>>>> >>>>>>> >>>>>>> everything sized >>>>>>>> correctly, but handling topology and iteration for the residual >>>>>>>> calculation >>>>>>>> would be completely >>>>>>>> up to you. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> It would be really great to define the initial data layout using IS >>>>>>> to >>>>>>> define the equation numbers of the splits, while letting the new DM >>>>>>> stuff >>>>>>> handle creating all the nested splits given command line options. Is >>>>>>> something like this possible? >>>>>>> >>>>>>> >>>>>>> This is just what I say above. Is something not working? >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>>> >>>>>>> Quoting Matthew Knepley : >>>>>>> >>>>>>> On Mon, Aug 13, 2012 at 4:06 PM, Colin McAuliffe < >>>>>>> cjm2176 at columbia.edu >>>>>>> >>>>>>> >wrote: >>>>>>>> >>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> You just have to define the data layout in a PetscSection and >>>>>>>>>> attach >>>>>>>>>> it >>>>>>>>>> to >>>>>>>>>> the DM with DMSetDefaultSection. >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Does use of the PetscSection mean that it is neccesary to define a >>>>>>>>>> >>>>>>>>> DMMesh? >>>>>>>>> In other >>>>>>>>> words is there a way to create the data layout for the physics >>>>>>>>> without >>>>>>>>> having to specify >>>>>>>>> any information about the mesh? >>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, just call DMSetDefaultSection(). However, note that you will >>>>>>>>> have >>>>>>>>> >>>>>>>> everything sized >>>>>>>> correctly, but handling topology and iteration for the residual >>>>>>>> calculation >>>>>>>> would be completely >>>>>>>> up to you. >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Quoting Matthew Knepley : >>>>>>>>> >>>>>>>>> On Thu, Aug 9, 2012 at 10:22 AM, Dmitry Karpeev < >>>>>>>>> karpeev at mcs.anl.gov> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Aug 9, 2012 at 10:02 AM, Colin McAuliffe < >>>>>>>>>> >>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>> >wrote: >>>>>>>>>>> >>>>>>>>>>> Sanjay, thanks for the reply but I am avoiding using blocked >>>>>>>>>>> format >>>>>>>>>>> >>>>>>>>>>> since >>>>>>>>>>> >>>>>>>>>>>> my problem has 10 dofs per node but only has either dofs 1-3 or >>>>>>>>>>>> 4-10 >>>>>>>>>>>> active >>>>>>>>>>>> on a particular node. If I use block the equations I run out of >>>>>>>>>>>> memory >>>>>>>>>>>> pretty quickly on my machine but can get to reasonable sized >>>>>>>>>>>> problems >>>>>>>>>>>> with >>>>>>>>>>>> the unblocked format. >>>>>>>>>>>> >>>>>>>>>>>> Matt, sorry I am not getting this, but I am still not sure how >>>>>>>>>>>> the >>>>>>>>>>>> DM >>>>>>>>>>>> interface works. I can see in the function >>>>>>>>>>>> PCFieldSplitSetDefaults >>>>>>>>>>>> that >>>>>>>>>>>> there is an initial call to DMCreateFieldDecomposition and >>>>>>>>>>>> subsequent >>>>>>>>>>>> calls >>>>>>>>>>>> to DMCreateSubDM based on the command line options. What I am >>>>>>>>>>>> missing >>>>>>>>>>>> is >>>>>>>>>>>> how the first call to DMCreateFieldDecomposition is able to >>>>>>>>>>>> figure >>>>>>>>>>>> out >>>>>>>>>>>> which equations belong to which field just from command line info >>>>>>>>>>>> such >>>>>>>>>>>> as >>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0. Where/how are the fields 2 and 0 >>>>>>>>>>>> defined? >>>>>>>>>>>> >>>>>>>>>>>> This might change slightly in the near future in petsc-dev to >>>>>>>>>>>> allow >>>>>>>>>>>> >>>>>>>>>>>> one to >>>>>>>>>>>> >>>>>>>>>>> define splits using named fields. >>>>>>>>>>> In any event, there has to be DM support to implement the >>>>>>>>>>> decompositions >>>>>>>>>>> over a particular mesh/problem over that mesh. >>>>>>>>>>> With DMDA you can essentially get combinations of strided fields >>>>>>>>>>> in >>>>>>>>>>> each >>>>>>>>>>> split. DMCOMPOSITE allows you >>>>>>>>>>> to pull out combinations of the subproblems that were put in >>>>>>>>>>> there to >>>>>>>>>>> begin with. If you have your own mesh, you have to write >>>>>>>>>>> a DM implementation around it to expose the available fields. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> No, No, No. You do not have to write a DM implementation. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> You just have to define the data layout in a PetscSection and >>>>>>>>>> attach >>>>>>>>>> it >>>>>>>>>> to >>>>>>>>>> the DM with DMSetDefaultSection. >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dmitry. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Aug 9, 2012 at 9:21 AM, Sanjay Govindjee < >>>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Colin, >>>>>>>>>>>>> >>>>>>>>>>>>> I you block the equations in FEAP, then the restrained BCs >>>>>>>>>>>>> are >>>>>>>>>>>>> >>>>>>>>>>>>> 'included' in assembled PETSc matrix (these dofs have rows that >>>>>>>>>>>>>> are >>>>>>>>>>>>>> zero >>>>>>>>>>>>>> modulo a value of unity on the diagonal and the restrained >>>>>>>>>>>>>> value >>>>>>>>>>>>>> on >>>>>>>>>>>>>> the >>>>>>>>>>>>>> right-hand side). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, this is not necessary with the DM interface. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -sg >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 8/9/12 8:41 AM, Colin McAuliffe wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From what I can gather from the petsc-dev source it looks like >>>>>>>>>>>>>> the >>>>>>>>>>>>>> >>>>>>>>>>>>>> commands in 4) will then generate the splits using strided >>>>>>>>>>>>>> blocks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The >>>>>>>>>>>>>>> problem with that is the fortran code I am using (FEAP) uses >>>>>>>>>>>>>>> petsc >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> assemble and solve the linear problem within its own nonlinear >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> time >>>>>>>>>>>>>>> stepping schemes. The linear problem that petsc solves already >>>>>>>>>>>>>>> has >>>>>>>>>>>>>>> boundary >>>>>>>>>>>>>>> conditions applied to it so petsc only sees the active >>>>>>>>>>>>>>> (unrestrained) >>>>>>>>>>>>>>> equations. So then in general fields can't be extracted from >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> active >>>>>>>>>>>>>>> equations using strided blocks and I am stuck with generating >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> index >>>>>>>>>>>>>>> sets defining the splits on my own. Will it still be possible >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> make >>>>>>>>>>>>>>> use >>>>>>>>>>>>>>> of the new DM functions in this case? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> FEAP website: >>>>>>>>>>>>>>> http://www.ce.berkeley.edu/************projects/feap/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Colin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Quoting Matthew Knepley : >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:51 PM, Matthew Knepley < >>>>>>>>>>>>>>> knepley at gmail.com >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Aug 8, 2012 at 10:23 PM, Colin McAuliffe < >>>>>>>>>>>>>>>> cjm2176 at columbia.edu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks all, regarding use of DM in 3.3, is the procedure >>>>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> the fields with PCFieldSplitSetIS and then use >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DMCreateFieldDecompositionDM >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> to create a new DM based from the new fields and the DM for >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> original >>>>>>>>>>>>>>>>>> problem? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1) Use petsc-dev >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2) PCFieldSplitSetIS() is independent. This allows you to >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>>> splits >>>>>>>>>>>>>>>>> however you want, but then recursive gets harder >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3) In 3.3., it uses DMCreateFieldDecompositionDM() to split >>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>> fields >>>>>>>>>>>>>>>>> apart at once >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 4) In petsc-dev, it uses DMCreateSubDM() which can split off >>>>>>>>>>>>>>>>> any >>>>>>>>>>>>>>>>> combination of fields, which from the command line is >>>>>>>>>>>>>>>>> something >>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -pc_fieldsplit_0_fields 2,0 -pc_fieldsplit_1_fields 1,3 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I should have shown recursive: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -fieldsplit_0_pc_type fieldsplit >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> will split 2,0 into two blocks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Colin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>>>>>>>> PhD Candidate >>>>>>>>>>>>>>>>>> Columbia University >>>>>>>>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>> experiments is infinitely more interesting than any results >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> experiments is infinitely more interesting than any results >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Sanjay Govindjee, PhD, PE >>>>>>>>>>>>>> Professor of Civil Engineering >>>>>>>>>>>>>> Vice Chair for Academic Affairs >>>>>>>>>>>>>> >>>>>>>>>>>>>> 779 Davis Hall >>>>>>>>>>>>>> Structural Engineering, Mechanics and Materials >>>>>>>>>>>>>> Department of Civil Engineering >>>>>>>>>>>>>> University of California >>>>>>>>>>>>>> Berkeley, CA 94720-1710 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Voice: +1 510 642 6060 >>>>>>>>>>>>>> FAX: +1 510 643 5264 >>>>>>>>>>>>>> s_g at berkeley.edu >>>>>>>>>>>>>> http://www.ce.berkeley.edu/~************sanjay >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>> http://berkeley.edu/~**sanjay**>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >< >>>>>>>>>>>>>> http://www.ce.berkeley.edu/~**********sanjay >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>> berkeley.edu/~sanjay >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ------------------------------************----------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>>>> which >>>>>>>>>>>>> their >>>>>>>>>>>>> experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>> Colin McAuliffe >>>>>>>>>>>> PhD Candidate >>>>>>>>>>>> Columbia University >>>>>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>> which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Colin McAuliffe >>>>>>>>> PhD Candidate >>>>>>>>> Columbia University >>>>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Colin McAuliffe >>>>>>> PhD Candidate >>>>>>> Columbia University >>>>>>> Department of Civil Engineering and Engineering Mechanics >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Colin McAuliffe >>>>> PhD Candidate >>>>> Columbia University >>>>> Department of Civil Engineering and Engineering Mechanics >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> -- >>> Colin McAuliffe >>> PhD Candidate >>> Columbia University >>> Department of Civil Engineering and Engineering Mechanics >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From jedbrown at mcs.anl.gov Mon Aug 27 21:34:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 21:34:11 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway wrote: > Unfortunately I can run backtraces since the code is running from python > and the -on_error_attach_debugger option has no effect when you're running > from python. > You can use -start_in_debugger, then type "c" (for "continue") at the gdb prompt. > > The petsc4py is initialized with: > import petsc4py > petsc4py.init(args=sys.argv) > > so other options are passed in just fine. > > I ran it with valgrind and got the following: > > ==15046== Jump to the invalid address stated on the next line > ==15046== at 0x0: ??? > ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) > ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) > ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) > ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) > ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) > ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) > ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) > ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) > ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in > /tmp/tmpGQGZda/sumb.so) > ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) > ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) > ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd > This looks like SNESShellSetSolve() did not work. Are you sure that the type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you called SNESShellSetSolve()? -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 21:52:28 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 22:52:28 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: I ran it though the debugger and came up with the following output: (gdb) backtrace #0 0x00000000 in ?? () #1 0xb5fd2832 in oursnesshellsolve (snes=0xbb03d30, x=0xbb2a4f0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/ftn-custom/zsnesshellf.c:13 #2 0xb6017009 in SNESSolve_Shell (snes=0xbb03d30) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c:169 #3 0xb60032e8 in SNESSolve (snes=0xbb03d30, b=0xb8c0610, x=0xbb2a4f0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 #4 0xb6041547 in SNESSolve_NGMRES (snes=0xb8c1f70) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c:259 #5 0xb60032e8 in SNESSolve (snes=0xb8c1f70, b=0xb8c0610, x=0xb8ba6f0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 #6 0xb5fd0e73 in snessolve_ (snes=0xb4d75a2c, b=0xb4d75c6c, x=0xb4d75c68, __ierr=0xbfffe338) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/ftn-custom/zsnesf.c:181 #7 0xb47791f3 in nksolver2 () at NKSolver2.F90:20 #8 0xb46f8f68 in solvestate () at solveState.F90:485 #9 0xb46f95c2 in solversteady () at solverSteady.f90:27 #10 0xb46f9257 in solver () at solver.F90:102 #11 0x0b8ba2c0 in ?? () #12 0xb464a222 in f2py_rout_sumb_solver () from /tmp/tmpTaGWsk/sumb.so #13 0xb46454e5 in fortran_call () from /tmp/tmpTaGWsk/sumb.so #14 0x0805fd6a in PyObject_Call () #15 0x080dd5b0 in PyEval_EvalFrameEx () #16 0x080dfbb2 in PyEval_EvalCodeEx () #17 0x08168f1f in ?? () #18 0x0805fd6a in PyObject_Call () #19 0x080dcbeb in PyEval_EvalFrameEx () #20 0x080dfbb2 in PyEval_EvalCodeEx () #21 0x08168e3c in ?? () #22 0x0805fd6a in PyObject_Call () #23 0x08067d5c in ?? () #24 0x0805fd6a in PyObject_Call () #25 0x080b3554 in ?? () #26 0x0805fd6a in PyObject_Call () #27 0x080dd5b0 in PyEval_EvalFrameEx () #28 0x080dfbb2 in PyEval_EvalCodeEx () #29 0x080dfca7 in PyEval_EvalCode () #30 0x080fd956 in PyRun_FileExFlags () #31 0x080fdbb2 in PyRun_SimpleFileExFlags () #32 0x0805b6d3 in Py_Main () #33 0x0805a8ab in main () I'm also 99% certain that the function in set correctly since the shell snes works just fine when it is called directly. (see some of the earlier code snippits). Its only when I'm trying to use it as the preconditioner for the another snes object (ngmres) I have issues. Gaetan On Mon, Aug 27, 2012 at 10:34 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway wrote: > >> Unfortunately I can run backtraces since the code is running from python >> and the -on_error_attach_debugger option has no effect when you're running >> from python. >> > > You can use -start_in_debugger, then type "c" (for "continue") at the gdb > prompt. > > >> >> The petsc4py is initialized with: >> import petsc4py >> petsc4py.init(args=sys.argv) >> >> so other options are passed in just fine. >> >> I ran it with valgrind and got the following: >> >> ==15046== Jump to the invalid address stated on the next line >> ==15046== at 0x0: ??? >> ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) >> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >> ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) >> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >> ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) >> ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) >> ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) >> ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) >> ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in >> /tmp/tmpGQGZda/sumb.so) >> ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) >> ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) >> ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd >> > > This looks like SNESShellSetSolve() did not work. Are you sure that the > type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you called > SNESShellSetSolve()? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 27 22:21:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 22:21:03 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 9:52 PM, Gaetan Kenway wrote: > I ran it though the debugger and came up with the following output: > (gdb) backtrace > #0 0x00000000 in ?? () > #1 0xb5fd2832 in oursnesshellsolve (snes=0xbb03d30, x=0xbb2a4f0) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/ftn-custom/zsnesshellf.c:13 > How are you calling SNESShellSetSolve(). The usage we expected was that you would call SNESSetFromOptions(snes,ierr) call SNESGetPC(snes,snespc,ierr) call SNESShellSetSolve(snespc,yourfunction,ierr) > #2 0xb6017009 in SNESSolve_Shell (snes=0xbb03d30) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c:169 > #3 0xb60032e8 in SNESSolve (snes=0xbb03d30, b=0xb8c0610, x=0xbb2a4f0) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 > #4 0xb6041547 in SNESSolve_NGMRES (snes=0xb8c1f70) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c:259 > #5 0xb60032e8 in SNESSolve (snes=0xb8c1f70, b=0xb8c0610, x=0xb8ba6f0) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 > #6 0xb5fd0e73 in snessolve_ (snes=0xb4d75a2c, b=0xb4d75c6c, x=0xb4d75c68, > __ierr=0xbfffe338) > at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/ftn-custom/zsnesf.c:181 > #7 0xb47791f3 in nksolver2 () at NKSolver2.F90:20 > #8 0xb46f8f68 in solvestate () at solveState.F90:485 > #9 0xb46f95c2 in solversteady () at solverSteady.f90:27 > #10 0xb46f9257 in solver () at solver.F90:102 > #11 0x0b8ba2c0 in ?? () > #12 0xb464a222 in f2py_rout_sumb_solver () from /tmp/tmpTaGWsk/sumb.so > #13 0xb46454e5 in fortran_call () from /tmp/tmpTaGWsk/sumb.so > #14 0x0805fd6a in PyObject_Call () > #15 0x080dd5b0 in PyEval_EvalFrameEx () > #16 0x080dfbb2 in PyEval_EvalCodeEx () > #17 0x08168f1f in ?? () > #18 0x0805fd6a in PyObject_Call () > #19 0x080dcbeb in PyEval_EvalFrameEx () > #20 0x080dfbb2 in PyEval_EvalCodeEx () > #21 0x08168e3c in ?? () > #22 0x0805fd6a in PyObject_Call () > #23 0x08067d5c in ?? () > #24 0x0805fd6a in PyObject_Call () > #25 0x080b3554 in ?? () > #26 0x0805fd6a in PyObject_Call () > #27 0x080dd5b0 in PyEval_EvalFrameEx () > #28 0x080dfbb2 in PyEval_EvalCodeEx () > #29 0x080dfca7 in PyEval_EvalCode () > #30 0x080fd956 in PyRun_FileExFlags () > #31 0x080fdbb2 in PyRun_SimpleFileExFlags () > #32 0x0805b6d3 in Py_Main () > #33 0x0805a8ab in main () > > I'm also 99% certain that the function in set correctly since the shell > snes works just fine when it is called directly. (see some of the earlier > code snippits). Its only when I'm trying to use it as the preconditioner > for the another snes object (ngmres) I have issues. > > Gaetan > > On Mon, Aug 27, 2012 at 10:34 PM, Jed Brown wrote: > >> On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway wrote: >> >>> Unfortunately I can run backtraces since the code is running from python >>> and the -on_error_attach_debugger option has no effect when you're running >>> from python. >>> >> >> You can use -start_in_debugger, then type "c" (for "continue") at the gdb >> prompt. >> >> >>> >>> The petsc4py is initialized with: >>> import petsc4py >>> petsc4py.init(args=sys.argv) >>> >>> so other options are passed in just fine. >>> >>> I ran it with valgrind and got the following: >>> >>> ==15046== Jump to the invalid address stated on the next line >>> ==15046== at 0x0: ??? >>> ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) >>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>> ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) >>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>> ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) >>> ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) >>> ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) >>> ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) >>> ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in >>> /tmp/tmpGQGZda/sumb.so) >>> ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) >>> ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) >>> ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd >>> >> >> This looks like SNESShellSetSolve() did not work. Are you sure that the >> type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you called >> SNESShellSetSolve()? >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 22:44:50 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 27 Aug 2012 23:44:50 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Following your suggestion I changed by calling routines to: call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) call SNESSetType(outer_snes, SNESNGMRES, ierr) call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) call snessetfromoptions(outer_snes, ierr) call SNESGetPC(outer_snes, psnes, ierr) call SNESSetType(psnes, SNESSHELL, ierr) call SNESShellSetSolve(psnes, psnes_func, ierr) And the exact same errors occur. I'm pretty sure it has something to do with combing the two snes objects. Both work just fine by themselves. Do some pointers get accidentally overwritten somehow? I also tried creating the 'psnes' first (with a SNESCreate) , and then using the SNESSetPC() instead but the same error happens. Gaetan On Mon, Aug 27, 2012 at 11:21 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 9:52 PM, Gaetan Kenway wrote: > >> I ran it though the debugger and came up with the following output: >> (gdb) backtrace >> #0 0x00000000 in ?? () >> #1 0xb5fd2832 in oursnesshellsolve (snes=0xbb03d30, x=0xbb2a4f0) at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/ftn-custom/zsnesshellf.c:13 >> > > How are you calling SNESShellSetSolve(). The usage we expected was that > you would > > call SNESSetFromOptions(snes,ierr) > call SNESGetPC(snes,snespc,ierr) > call SNESShellSetSolve(snespc,yourfunction,ierr) > > >> #2 0xb6017009 in SNESSolve_Shell (snes=0xbb03d30) at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c:169 >> #3 0xb60032e8 in SNESSolve (snes=0xbb03d30, b=0xb8c0610, x=0xbb2a4f0) at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >> #4 0xb6041547 in SNESSolve_NGMRES (snes=0xb8c1f70) at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c:259 >> #5 0xb60032e8 in SNESSolve (snes=0xb8c1f70, b=0xb8c0610, x=0xb8ba6f0) at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >> #6 0xb5fd0e73 in snessolve_ (snes=0xb4d75a2c, b=0xb4d75c6c, >> x=0xb4d75c68, __ierr=0xbfffe338) >> at >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/ftn-custom/zsnesf.c:181 >> #7 0xb47791f3 in nksolver2 () at NKSolver2.F90:20 >> #8 0xb46f8f68 in solvestate () at solveState.F90:485 >> #9 0xb46f95c2 in solversteady () at solverSteady.f90:27 >> #10 0xb46f9257 in solver () at solver.F90:102 >> #11 0x0b8ba2c0 in ?? () >> #12 0xb464a222 in f2py_rout_sumb_solver () from /tmp/tmpTaGWsk/sumb.so >> #13 0xb46454e5 in fortran_call () from /tmp/tmpTaGWsk/sumb.so >> #14 0x0805fd6a in PyObject_Call () >> #15 0x080dd5b0 in PyEval_EvalFrameEx () >> #16 0x080dfbb2 in PyEval_EvalCodeEx () >> #17 0x08168f1f in ?? () >> #18 0x0805fd6a in PyObject_Call () >> #19 0x080dcbeb in PyEval_EvalFrameEx () >> #20 0x080dfbb2 in PyEval_EvalCodeEx () >> #21 0x08168e3c in ?? () >> #22 0x0805fd6a in PyObject_Call () >> #23 0x08067d5c in ?? () >> #24 0x0805fd6a in PyObject_Call () >> #25 0x080b3554 in ?? () >> #26 0x0805fd6a in PyObject_Call () >> #27 0x080dd5b0 in PyEval_EvalFrameEx () >> #28 0x080dfbb2 in PyEval_EvalCodeEx () >> #29 0x080dfca7 in PyEval_EvalCode () >> #30 0x080fd956 in PyRun_FileExFlags () >> #31 0x080fdbb2 in PyRun_SimpleFileExFlags () >> #32 0x0805b6d3 in Py_Main () >> #33 0x0805a8ab in main () >> >> I'm also 99% certain that the function in set correctly since the shell >> snes works just fine when it is called directly. (see some of the earlier >> code snippits). Its only when I'm trying to use it as the preconditioner >> for the another snes object (ngmres) I have issues. >> >> Gaetan >> >> On Mon, Aug 27, 2012 at 10:34 PM, Jed Brown wrote: >> >>> On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway >> > wrote: >>> >>>> Unfortunately I can run backtraces since the code is running from >>>> python and the -on_error_attach_debugger option has no effect when you're >>>> running from python. >>>> >>> >>> You can use -start_in_debugger, then type "c" (for "continue") at the >>> gdb prompt. >>> >>> >>>> >>>> The petsc4py is initialized with: >>>> import petsc4py >>>> petsc4py.init(args=sys.argv) >>>> >>>> so other options are passed in just fine. >>>> >>>> I ran it with valgrind and got the following: >>>> >>>> ==15046== Jump to the invalid address stated on the next line >>>> ==15046== at 0x0: ??? >>>> ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) >>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>> ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) >>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>> ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) >>>> ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) >>>> ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) >>>> ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) >>>> ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in >>>> /tmp/tmpGQGZda/sumb.so) >>>> ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) >>>> ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) >>>> ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd >>>> >>> >>> This looks like SNESShellSetSolve() did not work. Are you sure that the >>> type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you called >>> SNESShellSetSolve()? >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperif at gmail.com Mon Aug 27 22:49:04 2012 From: sperif at gmail.com (Pierre-Yves Aquilanti) Date: Tue, 28 Aug 2012 11:49:04 +0800 Subject: [petsc-users] Matrix Market matrices in PETSc binary format - repository Message-ID: <38826BFC-957E-4DB1-8240-1F98299DA73A@gmail.com> Hello All, When developing software using PETSc, I had the need to experiment my codes using some tests matrices that I generally take from the University of Florida Sparse Matrix Collection. Those matrices are quite illustrative and help to check if my code works or allow me to bench it. The problem is that when in parallel, it is quite tricky to use a matrix in Matrix Market format as an input. And if I'm not mistaking, it is needed to convert the matrix from MM to PETSc binary format, which can take a lot time in some cases (I am converting dielFilterV3clx, it has been running for 162h and still not finished). For this reason, I am converting a bunch of matrices with a small piece of software, store them in some place and reuse them when needed. It allows me to avoid the burden or converting my matrices each time and also keep a small library of matrices for testing and benchmarking. I began to regroup the converted matrices on a github repo so others can use them : https://github.com/Perif/UF-PETSc-Matrices The program that I use for conversion is also available here : https://github.com/Perif/MatrixMarketToPETSc Please, feel free to comment to comment. It's the first time that I am sharing codes and data on the net and there might be some things to correct. I hope this could be useful to someone. Regards, PY From jedbrown at mcs.anl.gov Mon Aug 27 22:51:16 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 22:51:16 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: I think these function pointers should not be copied over, at least not in this way. I think Peter is working on a Fortran example to reproduce, but I think that if you apply the patch below, your calling sequence will work. diff --git a/src/snes/interface/snes.c b/src/snes/interface/snes.c --- a/src/snes/interface/snes.c +++ b/src/snes/interface/snes.c @@ -2463,9 +2463,6 @@ ierr = SNESSetApplicationContext(snes->pc,appctx);CHKERRQ(ierr); ierr = VecDestroy(&fpc);CHKERRQ(ierr); - /* copy the function pointers over */ - ierr = PetscObjectCopyFortranFunctionPointers((PetscObject)snes,(PetscObject)snes->pc);CHKERRQ(ierr); - /* default to 1 iteration */ ierr = SNESSetTolerances(snes->pc, 0.0, 0.0, 0.0, 1, snes->pc->max_funcs);CHKERRQ(ierr); ierr = SNESSetNormType(snes->pc, SNES_NORM_FINAL_ONLY);CHKERRQ(ierr); On Mon, Aug 27, 2012 at 10:44 PM, Gaetan Kenway wrote: > Following your suggestion I changed by calling routines to: > > call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) > call SNESSetType(outer_snes, SNESNGMRES, ierr) > call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) > call snessetfromoptions(outer_snes, ierr) > > call SNESGetPC(outer_snes, psnes, ierr) > call SNESSetType(psnes, SNESSHELL, ierr) > call SNESShellSetSolve(psnes, psnes_func, ierr) > > And the exact same errors occur. I'm pretty sure it has something to do > with combing the two snes objects. Both work just fine by themselves. Do > some pointers get accidentally overwritten somehow? I also tried creating > the 'psnes' first (with a SNESCreate) , and then using the SNESSetPC() > instead but the same error happens. > > Gaetan > > > On Mon, Aug 27, 2012 at 11:21 PM, Jed Brown wrote: > >> On Mon, Aug 27, 2012 at 9:52 PM, Gaetan Kenway wrote: >> >>> I ran it though the debugger and came up with the following output: >>> (gdb) backtrace >>> #0 0x00000000 in ?? () >>> #1 0xb5fd2832 in oursnesshellsolve (snes=0xbb03d30, x=0xbb2a4f0) at >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/ftn-custom/zsnesshellf.c:13 >>> >> >> How are you calling SNESShellSetSolve(). The usage we expected was that >> you would >> >> call SNESSetFromOptions(snes,ierr) >> call SNESGetPC(snes,snespc,ierr) >> call SNESShellSetSolve(snespc,yourfunction,ierr) >> >> >>> #2 0xb6017009 in SNESSolve_Shell (snes=0xbb03d30) at >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c:169 >>> #3 0xb60032e8 in SNESSolve (snes=0xbb03d30, b=0xb8c0610, x=0xbb2a4f0) >>> at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >>> #4 0xb6041547 in SNESSolve_NGMRES (snes=0xb8c1f70) at >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c:259 >>> #5 0xb60032e8 in SNESSolve (snes=0xb8c1f70, b=0xb8c0610, x=0xb8ba6f0) >>> at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >>> #6 0xb5fd0e73 in snessolve_ (snes=0xb4d75a2c, b=0xb4d75c6c, >>> x=0xb4d75c68, __ierr=0xbfffe338) >>> at >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/ftn-custom/zsnesf.c:181 >>> #7 0xb47791f3 in nksolver2 () at NKSolver2.F90:20 >>> #8 0xb46f8f68 in solvestate () at solveState.F90:485 >>> #9 0xb46f95c2 in solversteady () at solverSteady.f90:27 >>> #10 0xb46f9257 in solver () at solver.F90:102 >>> #11 0x0b8ba2c0 in ?? () >>> #12 0xb464a222 in f2py_rout_sumb_solver () from /tmp/tmpTaGWsk/sumb.so >>> #13 0xb46454e5 in fortran_call () from /tmp/tmpTaGWsk/sumb.so >>> #14 0x0805fd6a in PyObject_Call () >>> #15 0x080dd5b0 in PyEval_EvalFrameEx () >>> #16 0x080dfbb2 in PyEval_EvalCodeEx () >>> #17 0x08168f1f in ?? () >>> #18 0x0805fd6a in PyObject_Call () >>> #19 0x080dcbeb in PyEval_EvalFrameEx () >>> #20 0x080dfbb2 in PyEval_EvalCodeEx () >>> #21 0x08168e3c in ?? () >>> #22 0x0805fd6a in PyObject_Call () >>> #23 0x08067d5c in ?? () >>> #24 0x0805fd6a in PyObject_Call () >>> #25 0x080b3554 in ?? () >>> #26 0x0805fd6a in PyObject_Call () >>> #27 0x080dd5b0 in PyEval_EvalFrameEx () >>> #28 0x080dfbb2 in PyEval_EvalCodeEx () >>> #29 0x080dfca7 in PyEval_EvalCode () >>> #30 0x080fd956 in PyRun_FileExFlags () >>> #31 0x080fdbb2 in PyRun_SimpleFileExFlags () >>> #32 0x0805b6d3 in Py_Main () >>> #33 0x0805a8ab in main () >>> >>> I'm also 99% certain that the function in set correctly since the shell >>> snes works just fine when it is called directly. (see some of the earlier >>> code snippits). Its only when I'm trying to use it as the preconditioner >>> for the another snes object (ngmres) I have issues. >>> >>> Gaetan >>> >>> On Mon, Aug 27, 2012 at 10:34 PM, Jed Brown wrote: >>> >>>> On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway < >>>> kenway at utias.utoronto.ca> wrote: >>>> >>>>> Unfortunately I can run backtraces since the code is running from >>>>> python and the -on_error_attach_debugger option has no effect when you're >>>>> running from python. >>>>> >>>> >>>> You can use -start_in_debugger, then type "c" (for "continue") at the >>>> gdb prompt. >>>> >>>> >>>>> >>>>> The petsc4py is initialized with: >>>>> import petsc4py >>>>> petsc4py.init(args=sys.argv) >>>>> >>>>> so other options are passed in just fine. >>>>> >>>>> I ran it with valgrind and got the following: >>>>> >>>>> ==15046== Jump to the invalid address stated on the next line >>>>> ==15046== at 0x0: ??? >>>>> ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) >>>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>>> ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) >>>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>>> ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) >>>>> ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) >>>>> ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) >>>>> ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) >>>>> ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in >>>>> /tmp/tmpGQGZda/sumb.so) >>>>> ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) >>>>> ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) >>>>> ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd >>>>> >>>> >>>> This looks like SNESShellSetSolve() did not work. Are you sure that the >>>> type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you called >>>> SNESShellSetSolve()? >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Aug 27 23:13:39 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Tue, 28 Aug 2012 00:13:39 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Excellent! It finally runs without segfaulting. Now... to get it actually solve my problem.... :) Gaetan On Mon, Aug 27, 2012 at 11:51 PM, Jed Brown wrote: > I think these function pointers should not be copied over, at least not in > this way. I think Peter is working on a Fortran example to reproduce, but I > think that if you apply the patch below, your calling sequence will work. > > diff --git a/src/snes/interface/snes.c b/src/snes/interface/snes.c > --- a/src/snes/interface/snes.c > +++ b/src/snes/interface/snes.c > @@ -2463,9 +2463,6 @@ > ierr = SNESSetApplicationContext(snes->pc,appctx);CHKERRQ(ierr); > ierr = VecDestroy(&fpc);CHKERRQ(ierr); > > - /* copy the function pointers over */ > - ierr = > PetscObjectCopyFortranFunctionPointers((PetscObject)snes,(PetscObject)snes->pc);CHKERRQ(ierr); > - > /* default to 1 iteration */ > ierr = SNESSetTolerances(snes->pc, 0.0, 0.0, 0.0, 1, > snes->pc->max_funcs);CHKERRQ(ierr); > ierr = SNESSetNormType(snes->pc, SNES_NORM_FINAL_ONLY);CHKERRQ(ierr); > > > On Mon, Aug 27, 2012 at 10:44 PM, Gaetan Kenway wrote: > >> Following your suggestion I changed by calling routines to: >> >> call SNESCreate(SUMB_COMM_WORLD, outer_snes, ierr) >> call SNESSetType(outer_snes, SNESNGMRES, ierr) >> call SNESSetFunction(outer_snes, rVec, formFunction_snes, ctx2, ierr) >> call snessetfromoptions(outer_snes, ierr) >> >> call SNESGetPC(outer_snes, psnes, ierr) >> call SNESSetType(psnes, SNESSHELL, ierr) >> call SNESShellSetSolve(psnes, psnes_func, ierr) >> >> And the exact same errors occur. I'm pretty sure it has something to do >> with combing the two snes objects. Both work just fine by themselves. Do >> some pointers get accidentally overwritten somehow? I also tried creating >> the 'psnes' first (with a SNESCreate) , and then using the SNESSetPC() >> instead but the same error happens. >> >> Gaetan >> >> >> On Mon, Aug 27, 2012 at 11:21 PM, Jed Brown wrote: >> >>> On Mon, Aug 27, 2012 at 9:52 PM, Gaetan Kenway >> > wrote: >>> >>>> I ran it though the debugger and came up with the following output: >>>> (gdb) backtrace >>>> #0 0x00000000 in ?? () >>>> #1 0xb5fd2832 in oursnesshellsolve (snes=0xbb03d30, x=0xbb2a4f0) at >>>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/ftn-custom/zsnesshellf.c:13 >>>> >>> >>> How are you calling SNESShellSetSolve(). The usage we expected was that >>> you would >>> >>> call SNESSetFromOptions(snes,ierr) >>> call SNESGetPC(snes,snespc,ierr) >>> call SNESShellSetSolve(snespc,yourfunction,ierr) >>> >>> >>>> #2 0xb6017009 in SNESSolve_Shell (snes=0xbb03d30) at >>>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/shell/snesshell.c:169 >>>> #3 0xb60032e8 in SNESSolve (snes=0xbb03d30, b=0xb8c0610, x=0xbb2a4f0) >>>> at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >>>> #4 0xb6041547 in SNESSolve_NGMRES (snes=0xb8c1f70) at >>>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/impls/ngmres/snesngmres.c:259 >>>> #5 0xb60032e8 in SNESSolve (snes=0xb8c1f70, b=0xb8c0610, x=0xb8ba6f0) >>>> at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/snes.c:3536 >>>> #6 0xb5fd0e73 in snessolve_ (snes=0xb4d75a2c, b=0xb4d75c6c, >>>> x=0xb4d75c68, __ierr=0xbfffe338) >>>> at >>>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/snes/interface/ftn-custom/zsnesf.c:181 >>>> #7 0xb47791f3 in nksolver2 () at NKSolver2.F90:20 >>>> #8 0xb46f8f68 in solvestate () at solveState.F90:485 >>>> #9 0xb46f95c2 in solversteady () at solverSteady.f90:27 >>>> #10 0xb46f9257 in solver () at solver.F90:102 >>>> #11 0x0b8ba2c0 in ?? () >>>> #12 0xb464a222 in f2py_rout_sumb_solver () from /tmp/tmpTaGWsk/sumb.so >>>> #13 0xb46454e5 in fortran_call () from /tmp/tmpTaGWsk/sumb.so >>>> #14 0x0805fd6a in PyObject_Call () >>>> #15 0x080dd5b0 in PyEval_EvalFrameEx () >>>> #16 0x080dfbb2 in PyEval_EvalCodeEx () >>>> #17 0x08168f1f in ?? () >>>> #18 0x0805fd6a in PyObject_Call () >>>> #19 0x080dcbeb in PyEval_EvalFrameEx () >>>> #20 0x080dfbb2 in PyEval_EvalCodeEx () >>>> #21 0x08168e3c in ?? () >>>> #22 0x0805fd6a in PyObject_Call () >>>> #23 0x08067d5c in ?? () >>>> #24 0x0805fd6a in PyObject_Call () >>>> #25 0x080b3554 in ?? () >>>> #26 0x0805fd6a in PyObject_Call () >>>> #27 0x080dd5b0 in PyEval_EvalFrameEx () >>>> #28 0x080dfbb2 in PyEval_EvalCodeEx () >>>> #29 0x080dfca7 in PyEval_EvalCode () >>>> #30 0x080fd956 in PyRun_FileExFlags () >>>> #31 0x080fdbb2 in PyRun_SimpleFileExFlags () >>>> #32 0x0805b6d3 in Py_Main () >>>> #33 0x0805a8ab in main () >>>> >>>> I'm also 99% certain that the function in set correctly since the shell >>>> snes works just fine when it is called directly. (see some of the earlier >>>> code snippits). Its only when I'm trying to use it as the preconditioner >>>> for the another snes object (ngmres) I have issues. >>>> >>>> Gaetan >>>> >>>> On Mon, Aug 27, 2012 at 10:34 PM, Jed Brown wrote: >>>> >>>>> On Mon, Aug 27, 2012 at 7:31 PM, Gaetan Kenway < >>>>> kenway at utias.utoronto.ca> wrote: >>>>> >>>>>> Unfortunately I can run backtraces since the code is running from >>>>>> python and the -on_error_attach_debugger option has no effect when you're >>>>>> running from python. >>>>>> >>>>> >>>>> You can use -start_in_debugger, then type "c" (for "continue") at the >>>>> gdb prompt. >>>>> >>>>> >>>>>> >>>>>> The petsc4py is initialized with: >>>>>> import petsc4py >>>>>> petsc4py.init(args=sys.argv) >>>>>> >>>>>> so other options are passed in just fine. >>>>>> >>>>>> I ran it with valgrind and got the following: >>>>>> >>>>>> ==15046== Jump to the invalid address stated on the next line >>>>>> ==15046== at 0x0: ??? >>>>>> ==15046== by 0x95EF008: SNESSolve_Shell (snesshell.c:169) >>>>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>>>> ==15046== by 0x9619546: SNESSolve_NGMRES (snesngmres.c:259) >>>>>> ==15046== by 0x95DB2E7: SNESSolve (snes.c:3536) >>>>>> ==15046== by 0x95A8E72: snessolve_ (zsnesf.c:181) >>>>>> ==15046== by 0x9AEA302: nksolver2_ (NKSolver2.F90:20) >>>>>> ==15046== by 0x9A6A077: solvestate_ (solveState.F90:485) >>>>>> ==15046== by 0x9A6A6D1: solversteady_ (solverSteady.f90:27) >>>>>> ==15046== by 0x99BB1C1: f2py_rout_sumb_solver (in >>>>>> /tmp/tmpGQGZda/sumb.so) >>>>>> ==15046== by 0x99B6484: fortran_call (in /tmp/tmpGQGZda/sumb.so) >>>>>> ==15046== by 0x805FD69: PyObject_Call (in /usr/bin/python2.6) >>>>>> ==15046== Address 0x0 is not stack'd, malloc'd or (recently) free'd >>>>>> >>>>> >>>>> This looks like SNESShellSetSolve() did not work. Are you sure that >>>>> the type was set (e.g. SNESSetType() or SNESSetFromOptions()) before you >>>>> called SNESShellSetSolve()? >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 27 23:18:10 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 23:18:10 -0500 Subject: [petsc-users] Matrix Market matrices in PETSc binary format - repository In-Reply-To: <38826BFC-957E-4DB1-8240-1F98299DA73A@gmail.com> References: <38826BFC-957E-4DB1-8240-1F98299DA73A@gmail.com> Message-ID: On Mon, Aug 27, 2012 at 10:49 PM, Pierre-Yves Aquilanti wrote: > The problem is that when in parallel, it is quite tricky to use a matrix > in Matrix Market format as an input. And if I'm not mistaking, it is needed > to convert the matrix from MM to PETSc binary format, which can take a lot > time in some cases (I am converting dielFilterV3clx, it has been running > for 162h and still not finished). Right here, you don't preallocate: https://github.com/Perif/MatrixMarketToPETSc/blob/master/libs.c#L29 Preallocate (e.g. by making two passes over the input file) and it will take seconds to convert that file. You can write a very simple converter using scipy.io.mmread() and bin/pythonscripts/PetscBinaryIO.py. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 27 23:19:28 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 27 Aug 2012 23:19:28 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: On Mon, Aug 27, 2012 at 11:13 PM, Gaetan Kenway wrote: > Excellent! > > It finally runs without segfaulting. Now... to get it actually solve my > problem.... :) > Great. This is not a real fix, but the real fix will take a bit more effort. Let us know if you have any problems, otherwise we'll let you know when the proper fix is pushed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Aug 28 02:17:28 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 28 Aug 2012 09:17:28 +0200 Subject: [petsc-users] Null space correction depends on solver? Message-ID: <503C7088.1030507@tu-dresden.de> Part II of my never ending null space story :) Now I solve Poisson equation in 3D, discretized with FEM and using free boundaries. In the PETSc solver part, I added some test to check whether the solution is still orthogonal to the constant nullspace. The result is somehow hard to explain. Maybe one of you could spend one minute to give me some explanation what's going wrong and how to fix it. First, some code snippet: Vec rhs; Mat mat; .... Vec nsb; VecDuplicate(rhs, &nsb); VecSet(nsb, 1.0); MatNullSpace matNullspace; MatNullSpaceCreate(MPI::COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &matNullspace); MatNullSpaceRemove(matNullspace, rhs, PETSC_NULL); PetscScalar dot; VecDot(nsb, rhs, &dot); MSG("TEST 1 = %e\n", dot); KSP ksp; KSPCreate(MPI::COMM_WORLD, &kspMat11); KSPSetOperators(ksp, mat, mat, SAME_NONZERO_PATTERN); MatSetNullSpace(mat, matNullspace); KSPSetNullSpace(ksp, matNullspace); KSPSolve(mat, rhs, rhs); VecDot(nsb, rhs, &dot); MSG("TEST 2 = %e\n", dot); So, nothing really special. Now, I run the code with two different solvers: 1) -ksp_type fgmres -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_coarse Jacobi TEST 1 = 5.503931e-14 TEST 2 = -2.043290e+10 2) -ksp_type gmres -pc_type none TEST 1 = 5.503931e-14 TEST 2 = 3.195510e-09 How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. Thomas From bsmith at mcs.anl.gov Tue Aug 28 08:19:02 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Aug 2012 08:19:02 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: <503C7088.1030507@tu-dresden.de> References: <503C7088.1030507@tu-dresden.de> Message-ID: > How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. Are you sure the null space you provide is truly the null space of the matrix? Call MatNullSpaceTest(). GMRES uses left preconditioning by default which forces the null space into the Krylov space at each iteration. FGMRES uses right preconditioning and doesn't do anything to the null space because it is naturally in the Krylov space (since the result with fgmres contains the constants this is why I suspect the matrix actually doesn't have the null space you think it has). You can try running -ksp_type gmres -ksp_pc_side right my guess is that you will not get near zero for the dot. Barry On Aug 28, 2012, at 2:17 AM, Thomas Witkowski wrote: > Part II of my never ending null space story :) Now I solve Poisson equation in 3D, discretized with FEM and using free boundaries. In the PETSc solver part, I added some test to check whether the solution is still orthogonal to the constant nullspace. The result is somehow hard to explain. Maybe one of you could spend one minute to give me some explanation what's going wrong and how to fix it. First, some code snippet: > > Vec rhs; > Mat mat; > .... > Vec nsb; > VecDuplicate(rhs, &nsb); > VecSet(nsb, 1.0); > > MatNullSpace matNullspace; > MatNullSpaceCreate(MPI::COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &matNullspace); > MatNullSpaceRemove(matNullspace, rhs, PETSC_NULL); > > PetscScalar dot; > VecDot(nsb, rhs, &dot); > MSG("TEST 1 = %e\n", dot); > > KSP ksp; > KSPCreate(MPI::COMM_WORLD, &kspMat11); > KSPSetOperators(ksp, mat, mat, SAME_NONZERO_PATTERN); > > MatSetNullSpace(mat, matNullspace); > KSPSetNullSpace(ksp, matNullspace); > > KSPSolve(mat, rhs, rhs); > > VecDot(nsb, rhs, &dot); > MSG("TEST 2 = %e\n", dot); > > > So, nothing really special. Now, I run the code with two different solvers: > > 1) -ksp_type fgmres -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_coarse Jacobi > > TEST 1 = 5.503931e-14 > TEST 2 = -2.043290e+10 > > 2) -ksp_type gmres -pc_type none > TEST 1 = 5.503931e-14 > TEST 2 = 3.195510e-09 > > > How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. > > Thomas > From jedbrown at mcs.anl.gov Tue Aug 28 08:34:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 08:34:03 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: References: <503C7088.1030507@tu-dresden.de> Message-ID: On Tue, Aug 28, 2012 at 8:19 AM, Barry Smith wrote: > > How to interpret the results? For the first case, is the null space > information forwarded to the preconditioner? And even for using just gmres > without any preconditioner, the dot product is much closer to zero but > still far away. > > > Are you sure the null space you provide is truly the null space of the > matrix? Call MatNullSpaceTest(). > > GMRES uses left preconditioning by default which forces the null > space into the Krylov space at each iteration. FGMRES uses right > preconditioning and doesn't do anything to the null space because it is > naturally in the Krylov space (since the result with fgmres contains the > constants this is why I suspect the matrix actually doesn't have the null > space you think it has). You can try running -ksp_type gmres -ksp_pc_side > right my guess is that you will not get near zero for the dot. > Barry, it looks like null space removal is not implemented correctly for right preconditioning: diff --git a/src/ksp/ksp/examples/tutorials/ex29.c b/src/ksp/ksp/examples/tutorials/ex29.c --- a/src/ksp/ksp/examples/tutorials/ex29.c +++ b/src/ksp/ksp/examples/tutorials/ex29.c @@ -78,6 +78,12 @@ ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr); ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr); ierr = KSPGetRhs(ksp,&b);CHKERRQ(ierr); + { + PetscScalar dot; + ierr = VecSet(b,1.0);CHKERRQ(ierr); + ierr = VecDot(b,x,&dot);CHKERRQ(ierr); + ierr = PetscPrintf(PETSC_COMM_WORLD,"dot %g\n",dot);CHKERRQ(ierr); + } ierr = DMDestroy(&da);CHKERRQ(ierr); ierr = KSPDestroy(&ksp);CHKERRQ(ierr); $ ./ex29 -bc_type neumann -ksp_monitor_true_residual -ksp_test_null_space -mat_null_space_test_view -pc_type jacobi -ksp_pc_side left Constants are likely null vector|| A * 1/N || = 0 0 KSP preconditioned resid norm 1.135199672180e-01 true resid norm 2.319967097991e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.410876553096e-02 true resid norm 1.254974589275e-01 ||r(i)||/||b|| 5.409449945916e-01 2 KSP preconditioned resid norm 2.255575515098e-02 true resid norm 5.942041105213e-02 ||r(i)||/||b|| 2.561260937863e-01 3 KSP preconditioned resid norm 9.933310900089e-03 true resid norm 2.093328265899e-02 ||r(i)||/||b|| 9.023094627987e-02 4 KSP preconditioned resid norm 1.073259860881e-17 true resid norm 1.660357840786e-16 ||r(i)||/||b|| 7.156816328229e-16 dot 1.73472e-16 $ ./ex29 -bc_type neumann -ksp_monitor_true_residual -ksp_test_null_space -mat_null_space_test_view -pc_type jacobi -ksp_pc_side right Constants are likely null vector|| A * 1/N || = 0 0 KSP unpreconditioned resid norm 2.319967097991e-01 true resid norm 2.319967097991e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.218421389101e-01 true resid norm 1.218421389101e-01 ||r(i)||/||b|| 5.251890814123e-01 2 KSP unpreconditioned resid norm 5.897341285066e-02 true resid norm 5.897341285066e-02 ||r(i)||/||b|| 2.541993500758e-01 3 KSP unpreconditioned resid norm 2.064441965393e-02 true resid norm 2.064441965393e-02 ||r(i)||/||b|| 8.898582946200e-02 4 KSP unpreconditioned resid norm 8.019849301680e-17 true resid norm 5.101974804388e-17 ||r(i)||/||b|| 2.199158259100e-16 dot 0.0188443 It seems to be caused by this: #define KSP_RemoveNullSpace(ksp,y) ((ksp->nullsp && ksp->pc_side == PC_LEFT) ? MatNullSpaceRemove(ksp->nullsp,y,PETSC_NULL) : 0) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Aug 28 09:33:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 09:33:03 -0500 Subject: [petsc-users] Matrix Market matrices in PETSc binary format - repository In-Reply-To: References: <38826BFC-957E-4DB1-8240-1F98299DA73A@gmail.com> Message-ID: On Mon, Aug 27, 2012 at 11:18 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 10:49 PM, Pierre-Yves Aquilanti wrote: > >> The problem is that when in parallel, it is quite tricky to use a matrix >> in Matrix Market format as an input. And if I'm not mistaking, it is needed >> to convert the matrix from MM to PETSc binary format, which can take a lot >> time in some cases (I am converting dielFilterV3clx, it has been running >> for 162h and still not finished). > > > Right here, you don't preallocate: > > https://github.com/Perif/MatrixMarketToPETSc/blob/master/libs.c#L29 > > Preallocate (e.g. by making two passes over the input file) and it will > take seconds to convert that file. > > You can write a very simple converter using scipy.io.mmread() and > bin/pythonscripts/PetscBinaryIO.py. > It turns out there was a missing "self" in this code, fixed here http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/0d7ab44d6198 With that, these three lines of python will do the conversion for you import scipy.io, PetscBinaryIO A = scipy.io.mmread('dielFilterV3clx/dielFilterV3clx.mtx') PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('petscmatrix','w'), A) Make sure to set PETSC_ARCH for complex (for this matrix) or manually pass the precision/scalar-type flags. This runs in 30 seconds on my laptop. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brune at mcs.anl.gov Tue Aug 28 10:26:33 2012 From: brune at mcs.anl.gov (Peter Brune) Date: Tue, 28 Aug 2012 10:26:33 -0500 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: I've pushed a "real" fix to petsc-3.3 now. Using SNESShellSetSolve() on psnes should be enough now. - Peter On Mon, Aug 27, 2012 at 11:19 PM, Jed Brown wrote: > On Mon, Aug 27, 2012 at 11:13 PM, Gaetan Kenway wrote: > >> Excellent! >> >> It finally runs without segfaulting. Now... to get it actually solve my >> problem.... :) >> > > Great. This is not a real fix, but the real fix will take a bit more > effort. Let us know if you have any problems, otherwise we'll let you know > when the proper fix is pushed. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Aug 28 10:33:29 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 11:33:29 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices Message-ID: Dear All, I want to extract a sequential matrix from a MPIBAIJ matrix A with a dimension of 4n*4n. After assembly A, I use ISCreateStride to define the index by call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) And extract the matrix as call MatGetSubMatrices(Kmatem,1,isrow,isrow, & MAT_INITIAL_MATRIX,Kmatemseq) But it gives me a segmentation fault when I call ISCreateStride. Is there anything I did wrong? Thanks for any advice. Shu From knepley at gmail.com Tue Aug 28 10:36:26 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Aug 2012 10:36:26 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 10:33 AM, shu guo wrote: > Dear All, > > I want to extract a sequential matrix from a MPIBAIJ matrix A with a > dimension of 4n*4n. After assembly A, I use ISCreateStride to define > the index by > > call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > > And extract the matrix as > call MatGetSubMatrices(Kmatem,1,isrow,isrow, > & MAT_INITIAL_MATRIX,Kmatemseq) > > But it gives me a segmentation fault when I call ISCreateStride. Is > there anything I did wrong? Thanks for any advice. Run in the debugger, -start_in_debugger, and get a stack trace. Matt > > Shu > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Aug 28 10:37:54 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 10:37:54 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 10:33 AM, shu guo wrote: > Dear All, > > I want to extract a sequential matrix from a MPIBAIJ matrix A with a > dimension of 4n*4n. After assembly A, I use ISCreateStride to define > the index by > > call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > Fortran does not do type conversion without interfaces (which are about as much hassle as benefit). If you configured PETSc with non-default integer sizes, be sure to declare PetscInt zero,one instead of using numeric literals. Otherwise, run in valgrind to check that there isn't earlier corruption and run in a debugger to get a stack trace. > > And extract the matrix as > call MatGetSubMatrices(Kmatem,1,isrow,isrow, > & MAT_INITIAL_MATRIX,Kmatemseq) > > But it gives me a segmentation fault when I call ISCreateStride. Is > there anything I did wrong? Thanks for any advice. > > Shu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Aug 28 10:42:16 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 11:42:16 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: Hi Jed, I was thinking about this issue either. But when I create the matrix, I hardcoded as call MatCreateMPIBAIJ (PETSC_COMM_WORLD,4,32,32, & PETSC_DETERMINE, PETSC_DETERMINE, & PETSC_NULL_INTEGER, mr(np(246)), & PETSC_NULL_INTEGER, mr(np(247)), & Kmatem, ierr) And it works good. Does that mean this should not be a concern as you mentioned? Thanks. Shu On Tue, Aug 28, 2012 at 11:37 AM, Jed Brown wrote: > On Tue, Aug 28, 2012 at 10:33 AM, shu guo wrote: >> >> Dear All, >> >> I want to extract a sequential matrix from a MPIBAIJ matrix A with a >> dimension of 4n*4n. After assembly A, I use ISCreateStride to define >> the index by >> >> call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > > > Fortran does not do type conversion without interfaces (which are about as > much hassle as benefit). If you configured PETSc with non-default integer > sizes, be sure to declare PetscInt zero,one instead of using numeric > literals. > > Otherwise, run in valgrind to check that there isn't earlier corruption and > run in a debugger to get a stack trace. > >> >> >> And extract the matrix as >> call MatGetSubMatrices(Kmatem,1,isrow,isrow, >> & MAT_INITIAL_MATRIX,Kmatemseq) >> >> But it gives me a segmentation fault when I call ISCreateStride. Is >> there anything I did wrong? Thanks for any advice. >> >> Shu > > From jedbrown at mcs.anl.gov Tue Aug 28 10:44:24 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 10:44:24 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 10:42 AM, shu guo wrote: > Hi Jed, > > I was thinking about this issue either. But when I create the matrix, > I hardcoded as > > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,4,32,32, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & Kmatem, ierr) > > And it works good. Does that mean this should not be a concern as you > mentioned? > Probably, but it's still bad practice when using Fortran, and it will cause problems if you change the integer size later (e.g. to solve problems with more than 2B degrees of freedom). What does Valgrind say? > > Thanks. > > Shu > > > > On Tue, Aug 28, 2012 at 11:37 AM, Jed Brown wrote: > > On Tue, Aug 28, 2012 at 10:33 AM, shu guo wrote: > >> > >> Dear All, > >> > >> I want to extract a sequential matrix from a MPIBAIJ matrix A with a > >> dimension of 4n*4n. After assembly A, I use ISCreateStride to define > >> the index by > >> > >> call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > > > > > > Fortran does not do type conversion without interfaces (which are about > as > > much hassle as benefit). If you configured PETSc with non-default integer > > sizes, be sure to declare PetscInt zero,one instead of using numeric > > literals. > > > > Otherwise, run in valgrind to check that there isn't earlier corruption > and > > run in a debugger to get a stack trace. > > > >> > >> > >> And extract the matrix as > >> call MatGetSubMatrices(Kmatem,1,isrow,isrow, > >> & MAT_INITIAL_MATRIX,Kmatemseq) > >> > >> But it gives me a segmentation fault when I call ISCreateStride. Is > >> there anything I did wrong? Thanks for any advice. > >> > >> Shu > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Aug 28 10:48:09 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 11:48:09 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: I am using totalview instead of Valgrind. Where can I find the information of the stack trace?... Sorry about this easy question, I am really a beginner in programming... Shu On Tue, Aug 28, 2012 at 11:44 AM, Jed Brown wrote: > On Tue, Aug 28, 2012 at 10:42 AM, shu guo wrote: >> >> Hi Jed, >> >> I was thinking about this issue either. But when I create the matrix, >> I hardcoded as >> >> call MatCreateMPIBAIJ (PETSC_COMM_WORLD,4,32,32, >> & PETSC_DETERMINE, PETSC_DETERMINE, >> & PETSC_NULL_INTEGER, mr(np(246)), >> & PETSC_NULL_INTEGER, mr(np(247)), >> & Kmatem, ierr) >> >> And it works good. Does that mean this should not be a concern as you >> mentioned? > > > Probably, but it's still bad practice when using Fortran, and it will cause > problems if you change the integer size later (e.g. to solve problems with > more than 2B degrees of freedom). What does Valgrind say? > >> >> >> Thanks. >> >> Shu >> >> >> >> On Tue, Aug 28, 2012 at 11:37 AM, Jed Brown wrote: >> > On Tue, Aug 28, 2012 at 10:33 AM, shu guo wrote: >> >> >> >> Dear All, >> >> >> >> I want to extract a sequential matrix from a MPIBAIJ matrix A with a >> >> dimension of 4n*4n. After assembly A, I use ISCreateStride to define >> >> the index by >> >> >> >> call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) >> > >> > >> > Fortran does not do type conversion without interfaces (which are about >> > as >> > much hassle as benefit). If you configured PETSc with non-default >> > integer >> > sizes, be sure to declare PetscInt zero,one instead of using numeric >> > literals. >> > >> > Otherwise, run in valgrind to check that there isn't earlier corruption >> > and >> > run in a debugger to get a stack trace. >> > >> >> >> >> >> >> And extract the matrix as >> >> call MatGetSubMatrices(Kmatem,1,isrow,isrow, >> >> & MAT_INITIAL_MATRIX,Kmatemseq) >> >> >> >> But it gives me a segmentation fault when I call ISCreateStride. Is >> >> there anything I did wrong? Thanks for any advice. >> >> >> >> Shu >> > >> > > > From knepley at gmail.com Tue Aug 28 10:49:00 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Aug 2012 10:49:00 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 10:48 AM, shu guo wrote: > I am using totalview instead of Valgrind. Where can I find the > information of the stack trace?... Sorry about this easy question, I > am really a beginner in programming... > 'where' Matt > Shu > > On Tue, Aug 28, 2012 at 11:44 AM, Jed Brown wrote: > > On Tue, Aug 28, 2012 at 10:42 AM, shu guo wrote: > >> > >> Hi Jed, > >> > >> I was thinking about this issue either. But when I create the matrix, > >> I hardcoded as > >> > >> call MatCreateMPIBAIJ (PETSC_COMM_WORLD,4,32,32, > >> & PETSC_DETERMINE, PETSC_DETERMINE, > >> & PETSC_NULL_INTEGER, mr(np(246)), > >> & PETSC_NULL_INTEGER, mr(np(247)), > >> & Kmatem, ierr) > >> > >> And it works good. Does that mean this should not be a concern as you > >> mentioned? > > > > > > Probably, but it's still bad practice when using Fortran, and it will > cause > > problems if you change the integer size later (e.g. to solve problems > with > > more than 2B degrees of freedom). What does Valgrind say? > > > >> > >> > >> Thanks. > >> > >> Shu > >> > >> > >> > >> On Tue, Aug 28, 2012 at 11:37 AM, Jed Brown > wrote: > >> > On Tue, Aug 28, 2012 at 10:33 AM, shu guo > wrote: > >> >> > >> >> Dear All, > >> >> > >> >> I want to extract a sequential matrix from a MPIBAIJ matrix A with a > >> >> dimension of 4n*4n. After assembly A, I use ISCreateStride to define > >> >> the index by > >> >> > >> >> call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > >> > > >> > > >> > Fortran does not do type conversion without interfaces (which are > about > >> > as > >> > much hassle as benefit). If you configured PETSc with non-default > >> > integer > >> > sizes, be sure to declare PetscInt zero,one instead of using numeric > >> > literals. > >> > > >> > Otherwise, run in valgrind to check that there isn't earlier > corruption > >> > and > >> > run in a debugger to get a stack trace. > >> > > >> >> > >> >> > >> >> And extract the matrix as > >> >> call MatGetSubMatrices(Kmatem,1,isrow,isrow, > >> >> & MAT_INITIAL_MATRIX,Kmatemseq) > >> >> > >> >> But it gives me a segmentation fault when I call ISCreateStride. Is > >> >> there anything I did wrong? Thanks for any advice. > >> >> > >> >> Shu > >> > > >> > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Aug 28 10:56:29 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 11:56:29 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: Hi All, I think I made a mistake. It seems it crashes when I call the command of MatGetSubMatrices. There is an indication that saying, MatGetSubmatrices() is slightly different from C since the Fortran provides the array to hold the submatrix objects,while in C that array is allocated by the MatGetSubmatrices() What does this mean? I don't even need this command to extract the submatrix? Shu On Tue, Aug 28, 2012 at 11:33 AM, shu guo wrote: > Dear All, > > I want to extract a sequential matrix from a MPIBAIJ matrix A with a > dimension of 4n*4n. After assembly A, I use ISCreateStride to define > the index by > > call ISCreateStride(PETSC_COMM_WORLD,n,0,1,isrow,ierr) > > And extract the matrix as > call MatGetSubMatrices(Kmatem,1,isrow,isrow, > & MAT_INITIAL_MATRIX,Kmatemseq) > > But it gives me a segmentation fault when I call ISCreateStride. Is > there anything I did wrong? Thanks for any advice. > > Shu From jedbrown at mcs.anl.gov Tue Aug 28 11:03:51 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 11:03:51 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 10:56 AM, shu guo wrote: > Hi All, > > I think I made a mistake. It seems it crashes when I call the command > of MatGetSubMatrices. There is an indication that saying, > > MatGetSubmatrices() is slightly different from C since the > Fortran provides the array to hold the submatrix objects,while in C > that > array is allocated by the MatGetSubmatrices() > > What does this mean? I don't even need this command to extract the > submatrix? > It means that you have to allocate the array to hold the result. If this seems cumbersome to you, ask yourself why you chose to write in Fortran. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Aug 28 11:18:41 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 12:18:41 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: So I have to define this submatrix and extract is into this pre-set location? On Tue, Aug 28, 2012 at 12:03 PM, Jed Brown wrote: > On Tue, Aug 28, 2012 at 10:56 AM, shu guo wrote: >> >> Hi All, >> >> I think I made a mistake. It seems it crashes when I call the command >> of MatGetSubMatrices. There is an indication that saying, >> >> MatGetSubmatrices() is slightly different from C since the >> Fortran provides the array to hold the submatrix objects,while in C >> that >> array is allocated by the MatGetSubmatrices() >> >> What does this mean? I don't even need this command to extract the >> submatrix? > > > It means that you have to allocate the array to hold the result. > > If this seems cumbersome to you, ask yourself why you chose to write in > Fortran. ;-) From jedbrown at mcs.anl.gov Tue Aug 28 11:21:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 11:21:09 -0500 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: On Tue, Aug 28, 2012 at 11:18 AM, shu guo wrote: > So I have to define this submatrix and extract is into this pre-set > location? > No, you have to create an array to hold the Mat objects, e.g. Mat submats(10) > > On Tue, Aug 28, 2012 at 12:03 PM, Jed Brown wrote: > > On Tue, Aug 28, 2012 at 10:56 AM, shu guo wrote: > >> > >> Hi All, > >> > >> I think I made a mistake. It seems it crashes when I call the command > >> of MatGetSubMatrices. There is an indication that saying, > >> > >> MatGetSubmatrices() is slightly different from C since the > >> Fortran provides the array to hold the submatrix objects,while in C > >> that > >> array is allocated by the MatGetSubmatrices() > >> > >> What does this mean? I don't even need this command to extract the > >> submatrix? > > > > > > It means that you have to allocate the array to hold the result. > > > > If this seems cumbersome to you, ask yourself why you chose to write in > > Fortran. ;-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 28 11:33:15 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Aug 2012 11:33:15 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: References: <503C7088.1030507@tu-dresden.de> Message-ID: <8E75372B-4E63-4787-AB0E-CD20BF0923C8@mcs.anl.gov> On Aug 28, 2012, at 8:34 AM, Jed Brown wrote: > On Tue, Aug 28, 2012 at 8:19 AM, Barry Smith wrote: > > How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. > > > Are you sure the null space you provide is truly the null space of the matrix? Call MatNullSpaceTest(). > > GMRES uses left preconditioning by default which forces the null space into the Krylov space at each iteration. FGMRES uses right preconditioning and doesn't do anything to the null space because it is naturally in the Krylov space (since the result with fgmres contains the constants this is why I suspect the matrix actually doesn't have the null space you think it has). You can try running -ksp_type gmres -ksp_pc_side right my guess is that you will not get near zero for the dot. > > Barry, it looks like null space removal is not implemented correctly for right preconditioning: Is this possible to fix? Don't we need the null space of A B , which would require solving with B to get (from the null space of A) and in general we cannot solve with B, only apply it? Barry > > diff --git a/src/ksp/ksp/examples/tutorials/ex29.c b/src/ksp/ksp/examples/tutorials/ex29.c > --- a/src/ksp/ksp/examples/tutorials/ex29.c > +++ b/src/ksp/ksp/examples/tutorials/ex29.c > @@ -78,6 +78,12 @@ > ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr); > ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr); > ierr = KSPGetRhs(ksp,&b);CHKERRQ(ierr); > + { > + PetscScalar dot; > + ierr = VecSet(b,1.0);CHKERRQ(ierr); > + ierr = VecDot(b,x,&dot);CHKERRQ(ierr); > + ierr = PetscPrintf(PETSC_COMM_WORLD,"dot %g\n",dot);CHKERRQ(ierr); > + } > > ierr = DMDestroy(&da);CHKERRQ(ierr); > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > > $ ./ex29 -bc_type neumann -ksp_monitor_true_residual -ksp_test_null_space -mat_null_space_test_view -pc_type jacobi -ksp_pc_side left > Constants are likely null vector|| A * 1/N || = 0 > 0 KSP preconditioned resid norm 1.135199672180e-01 true resid norm 2.319967097991e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.410876553096e-02 true resid norm 1.254974589275e-01 ||r(i)||/||b|| 5.409449945916e-01 > 2 KSP preconditioned resid norm 2.255575515098e-02 true resid norm 5.942041105213e-02 ||r(i)||/||b|| 2.561260937863e-01 > 3 KSP preconditioned resid norm 9.933310900089e-03 true resid norm 2.093328265899e-02 ||r(i)||/||b|| 9.023094627987e-02 > 4 KSP preconditioned resid norm 1.073259860881e-17 true resid norm 1.660357840786e-16 ||r(i)||/||b|| 7.156816328229e-16 > dot 1.73472e-16 > $ ./ex29 -bc_type neumann -ksp_monitor_true_residual -ksp_test_null_space -mat_null_space_test_view -pc_type jacobi -ksp_pc_side right > Constants are likely null vector|| A * 1/N || = 0 > 0 KSP unpreconditioned resid norm 2.319967097991e-01 true resid norm 2.319967097991e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 1.218421389101e-01 true resid norm 1.218421389101e-01 ||r(i)||/||b|| 5.251890814123e-01 > 2 KSP unpreconditioned resid norm 5.897341285066e-02 true resid norm 5.897341285066e-02 ||r(i)||/||b|| 2.541993500758e-01 > 3 KSP unpreconditioned resid norm 2.064441965393e-02 true resid norm 2.064441965393e-02 ||r(i)||/||b|| 8.898582946200e-02 > 4 KSP unpreconditioned resid norm 8.019849301680e-17 true resid norm 5.101974804388e-17 ||r(i)||/||b|| 2.199158259100e-16 > dot 0.0188443 > > It seems to be caused by this: > > #define KSP_RemoveNullSpace(ksp,y) ((ksp->nullsp && ksp->pc_side == PC_LEFT) ? MatNullSpaceRemove(ksp->nullsp,y,PETSC_NULL) : 0) From thomas.witkowski at tu-dresden.de Tue Aug 28 12:25:56 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 28 Aug 2012 19:25:56 +0200 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: References: <503C7088.1030507@tu-dresden.de> Message-ID: <503CFF24.1040001@tu-dresden.de> Am 28.08.2012 15:19, schrieb Barry Smith: >> How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. > > Are you sure the null space you provide is truly the null space of the matrix? Call MatNullSpaceTest(). The test returns true! > GMRES uses left preconditioning by default which forces the null space into the Krylov space at each iteration. FGMRES uses right preconditioning and doesn't do anything to the null space because it is naturally in the Krylov space (since the result with fgmres contains the constants this is why I suspect the matrix actually doesn't have the null space you think it has). You can try running -ksp_type gmres -ksp_pc_side right my guess is that you will not get near zero for the dot. Any idea how to debug this problem? Lets concentrate on the gmres/no precon case first. Thomas > > Barry > > > > On Aug 28, 2012, at 2:17 AM, Thomas Witkowski wrote: > >> Part II of my never ending null space story :) Now I solve Poisson equation in 3D, discretized with FEM and using free boundaries. In the PETSc solver part, I added some test to check whether the solution is still orthogonal to the constant nullspace. The result is somehow hard to explain. Maybe one of you could spend one minute to give me some explanation what's going wrong and how to fix it. First, some code snippet: >> >> Vec rhs; >> Mat mat; >> .... >> Vec nsb; >> VecDuplicate(rhs,&nsb); >> VecSet(nsb, 1.0); >> >> MatNullSpace matNullspace; >> MatNullSpaceCreate(MPI::COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL,&matNullspace); >> MatNullSpaceRemove(matNullspace, rhs, PETSC_NULL); >> >> PetscScalar dot; >> VecDot(nsb, rhs,&dot); >> MSG("TEST 1 = %e\n", dot); >> >> KSP ksp; >> KSPCreate(MPI::COMM_WORLD,&kspMat11); >> KSPSetOperators(ksp, mat, mat, SAME_NONZERO_PATTERN); >> >> MatSetNullSpace(mat, matNullspace); >> KSPSetNullSpace(ksp, matNullspace); >> >> KSPSolve(mat, rhs, rhs); >> >> VecDot(nsb, rhs,&dot); >> MSG("TEST 2 = %e\n", dot); >> >> >> So, nothing really special. Now, I run the code with two different solvers: >> >> 1) -ksp_type fgmres -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_coarse Jacobi >> >> TEST 1 = 5.503931e-14 >> TEST 2 = -2.043290e+10 >> >> 2) -ksp_type gmres -pc_type none >> TEST 1 = 5.503931e-14 >> TEST 2 = 3.195510e-09 >> >> >> How to interpret the results? For the first case, is the null space information forwarded to the preconditioner? And even for using just gmres without any preconditioner, the dot product is much closer to zero but still far away. >> >> Thomas >> From greatgs2008 at gmail.com Tue Aug 28 12:40:49 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 28 Aug 2012 13:40:49 -0400 Subject: [petsc-users] ISCreateStride and MatGetSubmatrices In-Reply-To: References: Message-ID: Hi Jed, I solve the problem. It seems that that the array to hold the Mat is not necessary, at least I was not assign this value but it works fine. The mistake I made is that I forget the ierr using MatGetSubmatrices. It seems without this error indicator, it won't let me proceed. Thank you for your help. I will look into the tips on use Petsc in fortran. Shu On Tue, Aug 28, 2012 at 12:21 PM, Jed Brown wrote: > On Tue, Aug 28, 2012 at 11:18 AM, shu guo wrote: >> >> So I have to define this submatrix and extract is into this pre-set >> location? > > > No, you have to create an array to hold the Mat objects, e.g. > > Mat submats(10) > >> >> >> On Tue, Aug 28, 2012 at 12:03 PM, Jed Brown wrote: >> > On Tue, Aug 28, 2012 at 10:56 AM, shu guo wrote: >> >> >> >> Hi All, >> >> >> >> I think I made a mistake. It seems it crashes when I call the command >> >> of MatGetSubMatrices. There is an indication that saying, >> >> >> >> MatGetSubmatrices() is slightly different from C since the >> >> Fortran provides the array to hold the submatrix objects,while in >> >> C >> >> that >> >> array is allocated by the MatGetSubmatrices() >> >> >> >> What does this mean? I don't even need this command to extract the >> >> submatrix? >> > >> > >> > It means that you have to allocate the array to hold the result. >> > >> > If this seems cumbersome to you, ask yourself why you chose to write in >> > Fortran. ;-) > > From jedbrown at mcs.anl.gov Tue Aug 28 12:57:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 12:57:58 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: <503CFF24.1040001@tu-dresden.de> References: <503C7088.1030507@tu-dresden.de> <503CFF24.1040001@tu-dresden.de> Message-ID: On Tue, Aug 28, 2012 at 12:25 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > > GMRES uses left preconditioning by default which forces the null >> space into the Krylov space at each iteration. FGMRES uses right >> preconditioning and doesn't do anything to the null space because it is >> naturally in the Krylov space (since the result with fgmres contains the >> constants this is why I suspect the matrix actually doesn't have the null >> space you think it has). You can try running -ksp_type gmres -ksp_pc_side >> right my guess is that you will not get near zero for the dot. >> > Any idea how to debug this problem? Lets concentrate on the gmres/no > precon case first. Left preconditioning and anything using -pc_type none works correctly. It is only the case of right preconditioning that is incorrect (and needs to be fixed in PETSc). -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Aug 28 15:06:43 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 28 Aug 2012 22:06:43 +0200 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: References: <503C7088.1030507@tu-dresden.de> <503CFF24.1040001@tu-dresden.de> Message-ID: <503D24D3.7000703@tu-dresden.de> Am 28.08.2012 19:57, schrieb Jed Brown: > On Tue, Aug 28, 2012 at 12:25 PM, Thomas Witkowski > > wrote: > > > GMRES uses left preconditioning by default which forces > the null space into the Krylov space at each iteration. FGMRES > uses right preconditioning and doesn't do anything to the > null space because it is naturally in the Krylov space (since > the result with fgmres contains the constants this is why I > suspect the matrix actually doesn't have the null space you > think it has). You can try running -ksp_type gmres > -ksp_pc_side right my guess is that you will not get near > zero for the dot. > > Any idea how to debug this problem? Lets concentrate on the > gmres/no precon case first. > > > Left preconditioning and anything using -pc_type none works correctly. > It is only the case of right preconditioning that is incorrect (and > needs to be fixed in PETSc). Is the null space projected out at every iteration step? If so, can you give me a hint were I can find the corresponding line in source code? Thanks. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Aug 28 15:09:07 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 15:09:07 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: <503D24D3.7000703@tu-dresden.de> References: <503C7088.1030507@tu-dresden.de> <503CFF24.1040001@tu-dresden.de> <503D24D3.7000703@tu-dresden.de> Message-ID: On Tue, Aug 28, 2012 at 3:06 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Is the null space projected out at every iteration step? If so, can you > give me a hint were I can find the corresponding line in source code? > Thanks. It happens immediately after PCApply(), look at the end of petsc-private/kspimpl.h The problem is that it's not projected out for right preconditioning. We have to work out whether there is a way to support null space projection in general for right preconditioning, or whether that will require special support per-implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tisaac at ices.utexas.edu Tue Aug 28 15:13:13 2012 From: tisaac at ices.utexas.edu (Toby Isaac) Date: Tue, 28 Aug 2012 15:13:13 -0500 Subject: [petsc-users] =?utf-8?q?Interest_in_incorporating_extended_PCML?= =?utf-8?q?=3F?= Message-ID: <43cc28072e22b9595969a312acdf8da6@ices.utexas.edu> I've extended PCML to accept coordinates and then use them for some of the things ML does with coordinates (Zoltan repartitioning, auxiliary matrix based aggregation). I know that gamg does similar things, but I think what ML does is sufficiently different that this would be useful to some people. Do you think this is worth adding to petsc-dev? Cheers, Toby From jedbrown at mcs.anl.gov Tue Aug 28 15:14:50 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 28 Aug 2012 15:14:50 -0500 Subject: [petsc-users] Interest in incorporating extended PCML? In-Reply-To: <43cc28072e22b9595969a312acdf8da6@ices.utexas.edu> References: <43cc28072e22b9595969a312acdf8da6@ices.utexas.edu> Message-ID: On Tue, Aug 28, 2012 at 3:13 PM, Toby Isaac wrote: > > I've extended PCML to accept coordinates and then use them for some of the > things ML does with coordinates (Zoltan repartitioning, auxiliary matrix > based aggregation). I know that gamg does similar things, but I think what > ML does is sufficiently different that this would be useful to some people. > Do you think this is worth adding to petsc-dev? > Yes, we'd be happy to accept patches for that functionality. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pengxwang at hotmail.com Tue Aug 28 15:45:49 2012 From: pengxwang at hotmail.com (Roc Wang) Date: Tue, 28 Aug 2012 15:45:49 -0500 Subject: [petsc-users] variable declaration in Fortran90 Message-ID: Hello, I am developing Fortran90 code with PETSc. Several Modules are defined in the code. Such as Module MyParameter Implicit none #include "finclude/petscsys.h" #include "finclude/petscvec.h" #include "finclude/petscmat.h" #include "finclude/petscpc.h" #include "finclude/petscksp.h" #include "finclude/petscvec.h90" #include "finclude/petscmat.h90" Integer:: ... ... Real:: ... ... End Module MyParameter Module MySub Use Myparameter Contains ... ... Call MatSetType(A_Mat, MATAIJ, ierr) ... ... EndModule MySub Main Program Use MyParameter Use MySub ... ... ... ... End If I put all the modules in a single .F90 file, the compilation can be done successfully. However, If I put the modules in separated files, the following error information comes out: Error: Symbol 'mataij' at (1) has no IMPLICIT type. In addition, if I use PetscInt, PetscErrorCode to declare variables in Module MySub, the similar errors come out. How declare the variable with PETSc type in separate files? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Tue Aug 28 16:00:58 2012 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Tue, 28 Aug 2012 16:00:58 -0500 Subject: [petsc-users] variable declaration in Fortran90 In-Reply-To: References: Message-ID: >From what i understand, you should #include the corresponding petscdef files in the module and the petsc headers in the subroutine. petsc version installed on machine is petsc-3.2-p5 I use a single include file: #include "finclude/petscdef.h" in the module and in the subroutines (including the ones that are defined after contains in the module) and program. #include "finclude/petsc.h" you can use #include "finclude/petsc.h90" if you are using the things such as vecgetArrayF90 On Tue, Aug 28, 2012 at 3:45 PM, Roc Wang wrote: > Hello, > > > I am developing Fortran90 code with PETSc. Several Modules are defined > in the code. Such as > > Module MyParameter > Implicit none > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > #include "finclude/petscmat.h" > #include "finclude/petscpc.h" > #include "finclude/petscksp.h" > #include "finclude/petscvec.h90" > #include "finclude/petscmat.h90" > > Integer:: ... ... > Real:: ... ... > > > End Module MyParameter > > Module MySub > Use Myparameter > Contains > ... ... > Call MatSetType(A_Mat, MATAIJ, ierr) > ... ... > EndModule MySub > > > Main Program > Use MyParameter > Use MySub > ... ... > ... ... > End > > If I put all the modules in a single .F90 file, the compilation can be > done successfully. However, If I put the modules in separated files, the > following error information comes out: > > Error: Symbol 'mataij' at (1) has no IMPLICIT type. > > In addition, if I use PetscInt, PetscErrorCode to declare variables in > Module MySub, the similar errors come out. How declare the variable with > PETSc type in separate files? Thanks. > > > > > > > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Aug 29 00:56:18 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 29 Aug 2012 07:56:18 +0200 Subject: [petsc-users] Recompile PETSc after small changes Message-ID: <503DAF02.80600@tu-dresden.de> What's the most efficient way to recompile PETSc after making some small changes in very few files? Best regards, Thomas From aron.ahmadia at kaust.edu.sa Wed Aug 29 02:03:26 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 29 Aug 2012 08:03:26 +0100 Subject: [petsc-users] Recompile PETSc after small changes In-Reply-To: <503DAF02.80600@tu-dresden.de> References: <503DAF02.80600@tu-dresden.de> Message-ID: The CMake-generated Makefiles. A Sent from my iPhone On Aug 29, 2012, at 6:56 AM, Thomas Witkowski wrote: > What's the most efficient way to recompile PETSc after making some small changes in very few files? > > Best regards, > > Thomas -- ------------------------------ This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. From jedbrown at mcs.anl.gov Wed Aug 29 06:46:32 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 29 Aug 2012 06:46:32 -0500 Subject: [petsc-users] Recompile PETSc after small changes In-Reply-To: References: <503DAF02.80600@tu-dresden.de> Message-ID: To elaborate, if you have CMake on your machine, PETSc's configure will make it be used by default when you type "make". Those makefiles have dependencies written into them (unlike PETSc's legacy makefiles in the source tree) so recompilation after a small change should take about two seconds to compile that one file and link. On Wed, Aug 29, 2012 at 2:03 AM, Aron Ahmadia wrote: > The CMake-generated Makefiles. > > A > > Sent from my iPhone > > On Aug 29, 2012, at 6:56 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > > > What's the most efficient way to recompile PETSc after making some small > changes in very few files? > > > > Best regards, > > > > Thomas > > -- > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Wed Aug 29 09:59:29 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Wed, 29 Aug 2012 16:59:29 +0200 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options Message-ID: <-5971759389960931929@unknownmsgid> Hei! I was going through the Petsc user archives when I came across a discussion with Barry Smith regarding use of BoomerAMG through Petsc (http://lists.mcs.anl.gov/pipermail/petsc-users/2006-October/000692.html). What I did understand from that is some of the Hypre options cannot be accessed through Petsc. But options like Pilut are available as preconditioners or solvers depnding on ksptype. I am trying to use ILU smoothening to BoomerAMG (Hypre's manual lists the smoothers available and the function calls). The questions that remain after going through that discussion are: 1. Can a smoothening option in hypre's boomerAMG could be chosen at all using Petsc? 2. Could the function calls phrases used in Hypre itself be somehow employed in the Petsc code? Would be really grateful for any advice. Since I am using an IDE to build the code, non-command line option tricks would also be of great help. Arun Kamath NTNU Trondheim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 29 10:08:14 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Aug 2012 10:08:14 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <-5971759389960931929@unknownmsgid> References: <-5971759389960931929@unknownmsgid> Message-ID: On Wed, Aug 29, 2012 at 9:59 AM, Arun Kamath wrote: > Hei! > I was going through the Petsc user archives when I came across a > discussion with Barry Smith regarding use of BoomerAMG through Petsc > (http://lists.mcs.anl.gov/pipermail/petsc-users/2006-October/000692.html). > > What I did understand from that is some of the Hypre options cannot > be accessed through Petsc. But options like Pilut are available as > preconditioners or solvers depnding on ksptype. > I am trying to use ILU smoothening to BoomerAMG (Hypre's manual lists the > smoothers available and the function calls). The questions that remain > after going through that discussion are: > > 1. Can a smoothening option in hypre's boomerAMG could be chosen at all > using Petsc? > Yes (although I do not see ILU in their docs). You can see all the BoomerAMG options with -help. > 2. Could the function calls phrases used in Hypre itself be somehow > employed in the Petsc code? > > Would be really grateful for any advice. Since I am using an IDE to build > the code, non-command line option tricks would also be of great help. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsSetValue.html However, consider what you are doing. Changing code to run a different simulation, in the 21st century, is like hitching a horse to your Skoda. Matt > Arun Kamath > NTNU Trondheim > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 29 10:08:30 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 29 Aug 2012 10:08:30 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <-5971759389960931929@unknownmsgid> References: <-5971759389960931929@unknownmsgid> Message-ID: On Wed, Aug 29, 2012 at 9:59 AM, Arun Kamath wrote: > Hei! > I was going through the Petsc user archives when I came across a > discussion with Barry Smith regarding use of BoomerAMG through Petsc > (http://lists.mcs.anl.gov/pipermail/petsc-users/2006-October/000692.html). > > What I did understand from that is some of the Hypre options cannot > be accessed through Petsc. But options like Pilut are available as > preconditioners or solvers depnding on ksptype. > I am trying to use ILU smoothening to BoomerAMG (Hypre's manual lists the > smoothers available and the function calls). > HYPRE_BoomerAMGSetRelaxType, which is the relevant routine, does not support ILU. Hypre does not have an extensible system for smoothing so you can't add new smoothers without changing their library. > The questions that remain after going through that discussion are: > > 1. Can a smoothening option in hypre's boomerAMG could be chosen at all > using Petsc? > -pc_hypre_boomeramg_relax_type_all (or up/down) > > 2. Could the function calls phrases used in Hypre itself be somehow > employed in the Petsc code? > I don't know what you are asking for. > > > Would be really grateful for any advice. Since I am using an IDE to build > the code, non-command line option tricks would also be of great help. > You can set command line options with PetscOptionsSetValue(), but this is a slow way to experiment. I thought the purpose of an IDE was to make you more productive. Recompiling every time you change the solver is a good way to greatly reduce your productivity. Also, hard-coding configurations is a good way to make your software less useful later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Wed Aug 29 10:13:00 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 29 Aug 2012 16:13:00 +0100 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: References: <-5971759389960931929@unknownmsgid> Message-ID: >> Would be really grateful for any advice. Since I am using an IDE to build >> the code, non-command line option tricks would also be of great help. Any IDE worth the bandwidth needed to download it will have options for setting environment variables and command line arguments for different runs. A -- ------------------------------ This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. From bsmith at mcs.anl.gov Wed Aug 29 10:14:43 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Aug 2012 10:14:43 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <-5971759389960931929@unknownmsgid> References: <-5971759389960931929@unknownmsgid> Message-ID: <82CC5905-C38D-42C8-B28B-28ED9B9C6852@mcs.anl.gov> On Aug 29, 2012, at 9:59 AM, Arun Kamath wrote: > Hei! > I was going through the Petsc user archives when I came across a > discussion with Barry Smith regarding use of BoomerAMG through Petsc > (http://lists.mcs.anl.gov/pipermail/petsc-users/2006-October/000692.html). > > What I did understand from that is some of the Hypre options cannot > be accessed through Petsc. But options like Pilut are available as preconditioners or solvers depnding on ksptype. > I am trying to use ILU smoothening to BoomerAMG (Hypre's manual lists the smoothers available and the function calls). The questions that remain after going through that discussion are: > > 1. Can a smoothening option in hypre's boomerAMG could be chosen at all using Petsc? Currently all the BoomerAMG options that we DIRECTLY support can be found in the routine PCSetFromOptions_HYPRE_BoomerAMG() in the file src/ksp/pc/impls/hypre This was done a few years ago so may not include newer options added to BoomerAMG. If there are new options for BoomerAMG you would like added let us know and we can easily add them. But note that hypre is not composible in the same way as PETSc; you cannot mix parts of PILUT with BoomerAMG etc. > 2. Could the function calls phrases used in Hypre itself be somehow employed in the Petsc code? I have added PCHYPREGetSolver(PC pc,HYPRE_Solver *hsolver) to petsc-dev giving you access to the hypre solver so you can make function calls on it. Call this after you have called KSPSetFromOptions() (or SNESFromOptions() if you are using the nonlinear solvers). > > Would be really grateful for any advice. Since I am using an IDE to build the code, non-command line option tricks would also be of great help. This is a good place for a configuration file; the filename is passed to PetscInitialize() (or the default filename is petscrc). Here list the PETSc options one per line, you can add comments with # Thus even if you have many options it is easy to switch them around without recompiling the code. Barry > > Arun Kamath > NTNU Trondheim From pengxwang at hotmail.com Wed Aug 29 10:28:15 2012 From: pengxwang at hotmail.com (Roc Wang) Date: Wed, 29 Aug 2012 10:28:15 -0500 Subject: [petsc-users] variable declaration in Fortran90 In-Reply-To: References: , Message-ID: Thanks for the kindly reply. By saying petscdef files, do you mean petscXXXdef, where XXX is the corresponding component of PETSc, such as vec, mat, et. al.? I tried the following approach, but it still doesn't work. The errors come out since there are some PETSc variables defined in Module MyParameter. The compiler cannot find PETSc datatype. The conflict is that some PESTs variables have to defined in Module MyParameter, and other PETSc variables and PETSc subroutines are defined and called in another Module MySub. Any hints and suggestions would be appreciated very much. Thanks. !--------------the way I declared the PETSc files Module MyParameter Implicit none #include "finclude/petscsysdef.h" ! Put all petscXXXdef,h in the module #include "finclude/petscvecdef.h" #include "finclude/petscmatdef.h" #include "finclude/petscpcdef.h" #include "finclude/petsckspdef.h" Integer:: ... ... Real:: ... ... ! There are some PETSc variables defined in this Module End Module MyParameter Module MySub Use Myparameter Contains ... ... Subroutine MyPETSc_APP() #include "finclude/petscsys.h" #include "finclude/petscvec.h" #include "finclude/petscmat.h" #include "finclude/petscpc.h" #include "finclude/petscksp.h" #include "finclude/petscsys.h90" #include "finclude/petscmat.h90" #include "finclude/petscvec.h90" Implicit none ... ... Call MatSetType(A_Mat, MATAIJ, ierr) ... ... End Subroutine MyPETSc_APP ... ... EndModule MySub Main Program Use MyParameter Use MySub ... ... ... ... End Date: Tue, 28 Aug 2012 16:00:58 -0500 From: dharmareddy84 at gmail.com To: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] variable declaration in Fortran90 >From what i understand, you should #include the corresponding petscdef files in the module and the petsc headers in the subroutine. petsc version installed on machine is petsc-3.2-p5 I use a single include file: #include "finclude/petscdef.h" in the module and in the subroutines (including the ones that are defined after contains in the module) and program. #include "finclude/petsc.h" you can use #include "finclude/petsc.h90" if you are using the things such as vecgetArrayF90 On Tue, Aug 28, 2012 at 3:45 PM, Roc Wang wrote: Hello, I am developing Fortran90 code with PETSc. Several Modules are defined in the code. Such as Module MyParameter Implicit none #include "finclude/petscsys.h" #include "finclude/petscvec.h" #include "finclude/petscmat.h" #include "finclude/petscpc.h" #include "finclude/petscksp.h" #include "finclude/petscvec.h90" #include "finclude/petscmat.h90" Integer:: ... ... Real:: ... ... End Module MyParameter Module MySub Use Myparameter Contains ... ... Call MatSetType(A_Mat, MATAIJ, ierr) ... ... EndModule MySub Main Program Use MyParameter Use MySub ... ... ... ... End If I put all the modules in a single .F90 file, the compilation can be done successfully. However, If I put the modules in separated files, the following error information comes out: Error: Symbol 'mataij' at (1) has no IMPLICIT type. In addition, if I use PetscInt, PetscErrorCode to declare variables in Module MySub, the similar errors come out. How declare the variable with PETSc type in separate files? Thanks. -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 29 10:30:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Aug 2012 10:30:49 -0500 Subject: [petsc-users] variable declaration in Fortran90 In-Reply-To: References: Message-ID: On Wed, Aug 29, 2012 at 10:28 AM, Roc Wang wrote: > Thanks for the kindly reply. > > By saying petscdef files, do you mean petscXXXdef, where XXX is the > corresponding component of PETSc, such as vec, mat, et. al.? > > I tried the following approach, but it still doesn't work. The errors > come out since there are some PETSc variables defined in Module > MyParameter. The compiler cannot find PETSc datatype. The conflict is that > some PESTs variables have to defined in Module MyParameter, and other PETSc > variables and PETSc subroutines are defined and called in another Module > MySub. Any hints and suggestions would be appreciated very much. Thanks. > It is done this way here: http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex5f90.F.html and works fine. Matt > !--------------the way I declared the PETSc files > > Module MyParameter > Implicit none > #include "finclude/petscsysdef.h" ! Put all petscXXXdef,h in the > module > #include "finclude/petscvecdef.h" > #include "finclude/petscmatdef.h" > #include "finclude/petscpcdef.h" > #include "finclude/petsckspdef.h" > > > Integer:: ... ... > Real:: ... ... ! There are some PETSc variables defined in this > Module > > > End Module MyParameter > > Module MySub > Use Myparameter > Contains > ... ... > Subroutine MyPETSc_APP() > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > #include "finclude/petscmat.h" > #include "finclude/petscpc.h" > #include "finclude/petscksp.h" > > #include "finclude/petscsys.h90" > #include "finclude/petscmat.h90" > #include "finclude/petscvec.h90" > Implicit none > ... ... > Call MatSetType(A_Mat, MATAIJ, ierr) > ... ... > End Subroutine MyPETSc_APP > ... ... > EndModule MySub > > Main Program > Use MyParameter > Use MySub > ... ... > ... ... > End > > ------------------------------ > Date: Tue, 28 Aug 2012 16:00:58 -0500 > From: dharmareddy84 at gmail.com > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] variable declaration in Fortran90 > > From what i understand, you should #include the corresponding petscdef > files in the module and the petsc headers in the subroutine. > > petsc version installed on machine is petsc-3.2-p5 > I use a single include file: #include "finclude/petscdef.h" in the module > and in the subroutines (including the ones that are defined after contains > in the module) and program. > #include "finclude/petsc.h" > you can use #include "finclude/petsc.h90" if you are using the things > such as vecgetArrayF90 > > On Tue, Aug 28, 2012 at 3:45 PM, Roc Wang wrote: > > Hello, > > > I am developing Fortran90 code with PETSc. Several Modules are defined > in the code. Such as > > Module MyParameter > Implicit none > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > #include "finclude/petscmat.h" > #include "finclude/petscpc.h" > #include "finclude/petscksp.h" > #include "finclude/petscvec.h90" > #include "finclude/petscmat.h90" > > Integer:: ... ... > Real:: ... ... > > > End Module MyParameter > > Module MySub > Use Myparameter > Contains > ... ... > Call MatSetType(A_Mat, MATAIJ, ierr) > ... ... > EndModule MySub > > > Main Program > Use MyParameter > Use MySub > ... ... > ... ... > End > > If I put all the modules in a single .F90 file, the compilation can be > done successfully. However, If I put the modules in separated files, the > following error information comes out: > > Error: Symbol 'mataij' at (1) has no IMPLICIT type. > > In addition, if I use PetscInt, PetscErrorCode to declare variables in > Module MySub, the similar errors come out. How declare the variable with > PETSc type in separate files? Thanks. > > > > > > > > > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Wed Aug 29 11:50:56 2012 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 29 Aug 2012 11:50:56 -0500 Subject: [petsc-users] variable declaration in Fortran90 In-Reply-To: References: Message-ID: Hello Ron, I do not know how your are actually including the files and compiling. Have a look at the attached example illustrating petsc usage in Fortran, i tried to make the structure similar to the code you posted. You should be able to test it by (1) make test and (2) ./test Thanks Reddy On Wed, Aug 29, 2012 at 10:28 AM, Roc Wang wrote: > Thanks for the kindly reply. > > By saying petscdef files, do you mean petscXXXdef, where XXX is the > corresponding component of PETSc, such as vec, mat, et. al.? > > I tried the following approach, but it still doesn't work. The errors > come out since there are some PETSc variables defined in Module > MyParameter. The compiler cannot find PETSc datatype. The conflict is that > some PESTs variables have to defined in Module MyParameter, and other PETSc > variables and PETSc subroutines are defined and called in another Module > MySub. Any hints and suggestions would be appreciated very much. Thanks. > > !--------------the way I declared the PETSc files > > Module MyParameter > Implicit none > #include "finclude/petscsysdef.h" ! Put all petscXXXdef,h in the > module > #include "finclude/petscvecdef.h" > #include "finclude/petscmatdef.h" > #include "finclude/petscpcdef.h" > #include "finclude/petsckspdef.h" > > > Integer:: ... ... > Real:: ... ... ! There are some PETSc variables defined in this > Module > > > > End Module MyParameter > > Module MySub > Use Myparameter > Contains > ... ... > Subroutine MyPETSc_APP() > > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > #include "finclude/petscmat.h" > #include "finclude/petscpc.h" > #include "finclude/petscksp.h" > > #include "finclude/petscsys.h90" > #include "finclude/petscmat.h90" > #include "finclude/petscvec.h90" > Implicit none > > ... ... > Call MatSetType(A_Mat, MATAIJ, ierr) > ... ... > End Subroutine MyPETSc_APP > > ... ... > EndModule MySub > > Main Program > Use MyParameter > Use MySub > ... ... > ... ... > End > > ------------------------------ > Date: Tue, 28 Aug 2012 16:00:58 -0500 > From: dharmareddy84 at gmail.com > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] variable declaration in Fortran90 > > > From what i understand, you should #include the corresponding petscdef > files in the module and the petsc headers in the subroutine. > > petsc version installed on machine is petsc-3.2-p5 > I use a single include file: #include "finclude/petscdef.h" in the module > and in the subroutines (including the ones that are defined after contains > in the module) and program. > #include "finclude/petsc.h" > you can use #include "finclude/petsc.h90" if you are using the things > such as vecgetArrayF90 > > On Tue, Aug 28, 2012 at 3:45 PM, Roc Wang wrote: > > Hello, > > > I am developing Fortran90 code with PETSc. Several Modules are defined > in the code. Such as > > Module MyParameter > Implicit none > #include "finclude/petscsys.h" > #include "finclude/petscvec.h" > #include "finclude/petscmat.h" > #include "finclude/petscpc.h" > #include "finclude/petscksp.h" > #include "finclude/petscvec.h90" > #include "finclude/petscmat.h90" > > Integer:: ... ... > Real:: ... ... > > > End Module MyParameter > > Module MySub > Use Myparameter > Contains > ... ... > Call MatSetType(A_Mat, MATAIJ, ierr) > ... ... > EndModule MySub > > > Main Program > Use MyParameter > Use MySub > ... ... > ... ... > End > > If I put all the modules in a single .F90 file, the compilation can be > done successfully. However, If I put the modules in separated files, the > following error information comes out: > > Error: Symbol 'mataij' at (1) has no IMPLICIT type. > > In addition, if I use PetscInt, PetscErrorCode to declare variables in > Module MySub, the similar errors come out. How declare the variable with > PETSc type in separate files? Thanks. > > > > > > > > > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETScFortranExample.zip Type: application/zip Size: 1603 bytes Desc: not available URL: From kenway at utias.utoronto.ca Wed Aug 29 21:57:41 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Wed, 29 Aug 2012 22:57:41 -0400 Subject: [petsc-users] SNES Shell Problems In-Reply-To: References: Message-ID: Thanks Peter. I pulled the fix and it now works. Thank you (and Jed) for your help. Gaetan On Tue, Aug 28, 2012 at 11:26 AM, Peter Brune wrote: > I've pushed a "real" fix to petsc-3.3 now. Using SNESShellSetSolve() on > psnes should be enough now. > > - Peter > > > On Mon, Aug 27, 2012 at 11:19 PM, Jed Brown wrote: > >> On Mon, Aug 27, 2012 at 11:13 PM, Gaetan Kenway > > wrote: >> >>> Excellent! >>> >>> It finally runs without segfaulting. Now... to get it actually solve my >>> problem.... :) >>> >> >> Great. This is not a real fix, but the real fix will take a bit more >> effort. Let us know if you have any problems, otherwise we'll let you know >> when the proper fix is pushed. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 29 23:28:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 29 Aug 2012 23:28:57 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <82CC5905-C38D-42C8-B28B-28ED9B9C6852@mcs.anl.gov> References: <-5971759389960931929@unknownmsgid> <82CC5905-C38D-42C8-B28B-28ED9B9C6852@mcs.anl.gov> Message-ID: On Wed, Aug 29, 2012 at 10:14 AM, Barry Smith wrote: > This was done a few years ago so may not include newer options added > to BoomerAMG. If there are new options for BoomerAMG you would like added > let us know and we can easily add them. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/53de4ad33852 -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Thu Aug 30 02:53:51 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Thu, 30 Aug 2012 09:53:51 +0200 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options Message-ID: <5A81BE59-BF6E-4939-A4EF-12062A28AEED@gmail.com> Hei! Probably I was not very clear about my intentions/questions, which might have confused Jed and Matt somewhat, esp. the IDE part. Sorry about that. But Barry did get what I am doing. I am not hard-coding, there would be a control file for the user to choose between options to be executed at runtime. Thanks a lot guys, it is great to have developers so active on the User Forum. Greatly appreciated. From thomas.witkowski at tu-dresden.de Thu Aug 30 08:25:14 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 30 Aug 2012 15:25:14 +0200 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: References: <503C7088.1030507@tu-dresden.de> <503CFF24.1040001@tu-dresden.de> Message-ID: <503F69BA.3000406@tu-dresden.de> Am 28.08.2012 19:57, schrieb Jed Brown: > On Tue, Aug 28, 2012 at 12:25 PM, Thomas Witkowski > > wrote: > > > GMRES uses left preconditioning by default which forces > the null space into the Krylov space at each iteration. FGMRES > uses right preconditioning and doesn't do anything to the > null space because it is naturally in the Krylov space (since > the result with fgmres contains the constants this is why I > suspect the matrix actually doesn't have the null space you > think it has). You can try running -ksp_type gmres > -ksp_pc_side right my guess is that you will not get near > zero for the dot. > > Any idea how to debug this problem? Lets concentrate on the > gmres/no precon case first. > > > Left preconditioning and anything using -pc_type none works correctly. > It is only the case of right preconditioning that is incorrect (and > needs to be fixed in PETSc). If everything is fine for left preconditioning, can you please explain me what's going wrong in this minimal example. I patched src/ksp/ksp/examples/tutorials/ex4.c in the following way (just to make some test output): --- ex4-o.c 2012-08-30 15:14:52.852519625 +0200 +++ ex4.c 2012-08-30 15:16:33.304517903 +0200 @@ -169,9 +169,22 @@ ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &nullsp);CHKERRQ(ierr); ierr = KSPSetNullSpace(ksp, nullsp);CHKERRQ(ierr); + ierr = MatNullSpaceRemove(nullsp, x, PETSC_NULL);CHKERRQ(ierr); + Vec c, d; + ierr = VecDuplicate(x, &d);CHKERRQ(ierr); + ierr = VecDuplicate(x, &c);CHKERRQ(ierr); + ierr = VecSet(d, 1.0);CHKERRQ(ierr); + ierr = MatMult(A, d, c);CHKERRQ(ierr); + PetscScalar dot; + ierr = VecNorm(c, NORM_2, &dot);CHKERRQ(ierr); + ierr = PetscPrintf(PETSC_COMM_WORLD, "Norm of mat-mult = %e\n", dot); + ierr = VecDot(x, d, &dot);CHKERRQ(ierr); + ierr = PetscPrintf(PETSC_COMM_WORLD, "Dot product rhs = %e\n", dot); ierr = MatNullSpaceDestroy(&nullsp);CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSolve(ksp, b, x);CHKERRQ(ierr); + ierr = VecDot(b, d, &dot);CHKERRQ(ierr); + ierr = PetscPrintf(PETSC_COMM_WORLD, "Dot product sol = %e\n", dot); ierr = VecDestroy(&x);CHKERRQ(ierr); ierr = VecDestroy(&b);CHKERRQ(ierr); /* Solve physical system: I run this example with the following command: mpiexec -np 1 ./ex4 -gpu no -cpu -view no -solve -ksp_monitor_true_residual -ksp_type gmres -pc_type none The output is the following: Norm of mat-mult = 0.000000e+00 Dot product rhs = 0.000000e+00 0 KSP preconditioned resid norm 9.544889963521e-01 true resid norm 1.322454133866e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.183073265542e-01 true resid norm 1.104599312915e+00 ||r(i)||/||b|| 8.352647435007e-01 2 KSP preconditioned resid norm 1.770056136651e-01 true resid norm 9.322910914997e-01 ||r(i)||/||b|| 7.049704542680e-01 3 KSP preconditioned resid norm 5.212633308436e-02 true resid norm 9.168166919410e-01 ||r(i)||/||b|| 6.932691792200e-01 4 KSP preconditioned resid norm 2.714317700314e-02 true resid norm 9.157360122211e-01 ||r(i)||/||b|| 6.924520017526e-01 5 KSP preconditioned resid norm 1.718946122098e-03 true resid norm 9.153352646963e-01 ||r(i)||/||b|| 6.921489685398e-01 6 KSP preconditioned resid norm 2.794230282044e-16 true resid norm 9.153336506547e-01 ||r(i)||/||b|| 6.921477480502e-01 Dot product sol = 2.746001e+00 As you see, the constant vector is a member of the nullspace (and it should span it), the right hand side is orthogonal to the null space, but not the solution vector. Also the solver does not converge in the true residual norm. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 30 08:48:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 30 Aug 2012 08:48:41 -0500 Subject: [petsc-users] Null space correction depends on solver? In-Reply-To: <503F69BA.3000406@tu-dresden.de> References: <503C7088.1030507@tu-dresden.de> <503CFF24.1040001@tu-dresden.de> <503F69BA.3000406@tu-dresden.de> Message-ID: You seem to have the solution and right hand side vectors mixed up. On Aug 30, 2012 8:25 AM, "Thomas Witkowski" wrote: > Am 28.08.2012 19:57, schrieb Jed Brown: > > On Tue, Aug 28, 2012 at 12:25 PM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> >> GMRES uses left preconditioning by default which forces the null >>> space into the Krylov space at each iteration. FGMRES uses right >>> preconditioning and doesn't do anything to the null space because it is >>> naturally in the Krylov space (since the result with fgmres contains the >>> constants this is why I suspect the matrix actually doesn't have the null >>> space you think it has). You can try running -ksp_type gmres -ksp_pc_side >>> right my guess is that you will not get near zero for the dot. >>> >> Any idea how to debug this problem? Lets concentrate on the gmres/no >> precon case first. > > > Left preconditioning and anything using -pc_type none works correctly. It > is only the case of right preconditioning that is incorrect (and needs to > be fixed in PETSc). > > If everything is fine for left preconditioning, can you please explain me > what's going wrong in this minimal example. I patched > src/ksp/ksp/examples/tutorials/ex4.c in the following way (just to make > some test output): > > --- ex4-o.c 2012-08-30 15:14:52.852519625 +0200 > +++ ex4.c 2012-08-30 15:16:33.304517903 +0200 > @@ -169,9 +169,22 @@ > ierr = KSPSetOperators(ksp, A, A, > DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, > PETSC_NULL, &nullsp);CHKERRQ(ierr); > ierr = KSPSetNullSpace(ksp, nullsp);CHKERRQ(ierr); > + ierr = MatNullSpaceRemove(nullsp, x, PETSC_NULL);CHKERRQ(ierr); > + Vec c, d; > + ierr = VecDuplicate(x, &d);CHKERRQ(ierr); > + ierr = VecDuplicate(x, &c);CHKERRQ(ierr); > + ierr = VecSet(d, 1.0);CHKERRQ(ierr); > + ierr = MatMult(A, d, c);CHKERRQ(ierr); > + PetscScalar dot; > + ierr = VecNorm(c, NORM_2, &dot);CHKERRQ(ierr); > + ierr = PetscPrintf(PETSC_COMM_WORLD, "Norm of mat-mult = %e\n", dot); > + ierr = VecDot(x, d, &dot);CHKERRQ(ierr); > + ierr = PetscPrintf(PETSC_COMM_WORLD, "Dot product rhs = %e\n", dot); > ierr = MatNullSpaceDestroy(&nullsp);CHKERRQ(ierr); > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > ierr = KSPSolve(ksp, b, x);CHKERRQ(ierr); > + ierr = VecDot(b, d, &dot);CHKERRQ(ierr); > + ierr = PetscPrintf(PETSC_COMM_WORLD, "Dot product sol = %e\n", dot); > ierr = VecDestroy(&x);CHKERRQ(ierr); > ierr = VecDestroy(&b);CHKERRQ(ierr); > /* Solve physical system: > > > I run this example with the following command: > > mpiexec -np 1 ./ex4 -gpu no -cpu -view no -solve > -ksp_monitor_true_residual -ksp_type gmres -pc_type none > > The output is the following: > > Norm of mat-mult = 0.000000e+00 > Dot product rhs = 0.000000e+00 > 0 KSP preconditioned resid norm 9.544889963521e-01 true resid norm > 1.322454133866e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.183073265542e-01 true resid norm > 1.104599312915e+00 ||r(i)||/||b|| 8.352647435007e-01 > 2 KSP preconditioned resid norm 1.770056136651e-01 true resid norm > 9.322910914997e-01 ||r(i)||/||b|| 7.049704542680e-01 > 3 KSP preconditioned resid norm 5.212633308436e-02 true resid norm > 9.168166919410e-01 ||r(i)||/||b|| 6.932691792200e-01 > 4 KSP preconditioned resid norm 2.714317700314e-02 true resid norm > 9.157360122211e-01 ||r(i)||/||b|| 6.924520017526e-01 > 5 KSP preconditioned resid norm 1.718946122098e-03 true resid norm > 9.153352646963e-01 ||r(i)||/||b|| 6.921489685398e-01 > 6 KSP preconditioned resid norm 2.794230282044e-16 true resid norm > 9.153336506547e-01 ||r(i)||/||b|| 6.921477480502e-01 > Dot product sol = 2.746001e+00 > > As you see, the constant vector is a member of the nullspace (and it > should span it), the right hand side is orthogonal to the null space, but > not the solution vector. Also the solver does not converge in the true > residual norm. > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Thu Aug 30 10:58:48 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Thu, 30 Aug 2012 17:58:48 +0200 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options Message-ID: <4BAA7A16-EABD-4665-B20E-E070D22B4BA9@gmail.com> Hei! As per Barry's instructions, to introduce the "relax type" routine for the BoomerAMG solver I used: ierr=PCSetFromOptions_HYPRE_BoomerAMG(pc); (as per function definition in hypre.c) and on building : 'PCSetFromOptions_HYPRE_BoomerAMG' was not declared in this scope So am I probably missing a header file? As of now I have only . Or am I calling it wrong? Thanks. From knepley at gmail.com Thu Aug 30 11:11:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 30 Aug 2012 11:11:57 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <4BAA7A16-EABD-4665-B20E-E070D22B4BA9@gmail.com> References: <4BAA7A16-EABD-4665-B20E-E070D22B4BA9@gmail.com> Message-ID: On Thu, Aug 30, 2012 at 10:58 AM, Arun Kamath wrote: > Hei! > As per Barry's instructions, to introduce the "relax type" routine for the > BoomerAMG solver I used: > > ierr=PCSetFromOptions_HYPRE_BoomerAMG(pc); (as per function > definition in hypre.c) > No, you just call PCSetFrmOptions(pc) and it will dispatch to the correct function. Matt > and on building : > > 'PCSetFromOptions_HYPRE_BoomerAMG' was not declared in this scope > > So am I probably missing a header file? As of now I have only . > Or am I calling it wrong? > Thanks. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From arun.kamath.m at gmail.com Thu Aug 30 11:49:11 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Thu, 30 Aug 2012 18:49:11 +0200 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options Message-ID: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> Ok.. but as I follow it up with : ierr=PCSetFromOptions(pc); ierr= PetscOptionsSetValue("-pc_hypre_boomeramg_relax_type_all","0"); CHKERRQ(ierr); , on execution, the option is not recognized, as in: WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-pc_hypre_boomeramg_relax_type_all value: 0 is returned. I used the value of zero to employ the Jacobi smoother defined in the Hypre Ref manual. So could you please let me know what could I do make it work? (Since I want to access all solvers and options through a 'control'/ 'config' file, I can't ask for the option at the the command prompt ) Thanks. From knepley at gmail.com Thu Aug 30 11:54:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 30 Aug 2012 11:54:44 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> References: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> Message-ID: On Thu, Aug 30, 2012 at 11:49 AM, Arun Kamath wrote: > Ok.. but as I follow it up with : > ierr=PCSetFromOptions(pc); > ierr= > PetscOptionsSetValue("-pc_hypre_boomeramg_relax_type_all","0"); > CHKERRQ(ierr); , > 1) Form the code you would do ierr= PetscOptionsSetValue("-pc_hypre_boomeramg_relax_type_all","0"); CHKERRQ(ierr); , ierr=PCSetFromOptions(pc); 2) Why would you do that? I thought you were going to use an options file as Barry suggested? Matt on execution, the option is not recognized, as in: > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-pc_hypre_boomeramg_relax_type_all value: 0 > > is returned. > > I used the value of zero to employ the Jacobi smoother defined in the > Hypre Ref manual. > > So could you please let me know what could I do make it work? > (Since I want to access all solvers and options through a 'control'/ > 'config' file, I can't ask for the option at the the command prompt ) > Thanks. > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 30 11:55:30 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 30 Aug 2012 11:55:30 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> References: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> Message-ID: On Thu, Aug 30, 2012 at 11:49 AM, Arun Kamath wrote: > Ok.. but as I follow it up with : > ierr=PCSetFromOptions(pc); > ierr= > PetscOptionsSetValue("-pc_hypre_boomeramg_relax_type_all","0"); > CHKERRQ(ierr); , > You have to set the value _before_ calling PCSetFromOptions. Note that you generally don't have to call PCSetFromOptions directly, just call KSPSetFromOptions. > > on execution, the option is not recognized, as in: > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-pc_hypre_boomeramg_relax_type_all value: 0 > > is returned. > > I used the value of zero to employ the Jacobi smoother defined in the > Hypre Ref manual. > We think that cryptic non-sequential numbers are a terrible interface so this option takes a string. $ ./ex2 -pc_type hypre -help |grep relax_ -pc_hypre_boomeramg_relax_type_all (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi Gaussian-elimination (None) -pc_hypre_boomeramg_relax_type_down (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi Gaussian-elimination (None) -pc_hypre_boomeramg_relax_type_up (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi Gaussian-elimination (None) -pc_hypre_boomeramg_relax_type_coarse (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi Gaussian-elimination (None) -pc_hypre_boomeramg_relax_weight_all <1>: Relaxation weight for all levels (0 = hypre estimates, -k = determined with k CG steps) (None) -pc_hypre_boomeramg_relax_weight_level <1>: Set the relaxation weight for a particular level (weight,level) (None) -pc_hypre_boomeramg_outer_relax_weight_all <1>: Outer relaxation weight for all levels ( -k = determined with k CG steps) (None) -pc_hypre_boomeramg_outer_relax_weight_level <1>: Set the outer relaxation weight for a particular level (weight,level) (None) > > So could you please let me know what could I do make it work? > (Since I want to access all solvers and options through a 'control'/ > 'config' file, Why not put it all in an options file? > I can't ask for the option at the the command prompt ) > Thanks. > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 30 12:04:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 30 Aug 2012 12:04:43 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: References: <9E105176-985A-4C1F-B786-5A74EAEB17DD@gmail.com> Message-ID: On Thu, Aug 30, 2012 at 11:55 AM, Jed Brown wrote: > We think that cryptic non-sequential numbers are a terrible interface so > this option takes a string. > > $ ./ex2 -pc_type hypre -help |grep relax_ > -pc_hypre_boomeramg_relax_type_all (choose one > of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi > backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi > Gaussian-elimination (None) > The list above was incomplete because some values were being hidden. Fixed here. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/f852858bf195 -pc_hypre_boomeramg_relax_type_all (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi symmetric-SOR/Jacobi l1scaled-SOR/Jacobi Gaussian-elimination CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None) -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Thu Aug 30 15:51:52 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Thu, 30 Aug 2012 16:51:52 -0400 Subject: [petsc-users] mpiaijperm and mpiaijcrl Message-ID: Hi I was wondering if anyone had any experience with using these new matrix formats in PETSc. I have a block aij matrix (with block size 5) and tried converting to either of these types and it just trashed memory for 10 minutes and I then killed it. Matrix assemble is 13 seconds for reference. I then tried to construct the matrix directly with "MatCreateMPIAIJPERM" or "MatCreateMPIAIJCRL". Neither of these functions are wrapped in Fortran. Also the documentation for "MatCreateMPIAIJCRL" is extremely confusing as it says that comm is set to PETSC_COMM_SELF which defeats the purpose of a MPI matrix. Therefor, two questions: 1. Have any users seen a performance benefit of either of these formats (in parallel)? 2. How does one actually efficiently create such matrices? (from Fortran) On a side note, is it possible somehow to indicate on the manual pages all the functions that are not actually wrapped in Fortran? It would make it easier to know which one you can and can't use instead of having the compiler tell you it can't find a symbol. Thanks, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Aug 30 18:08:45 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 30 Aug 2012 18:08:45 -0500 Subject: [petsc-users] Memory Accumulation Message-ID: <503FF27D.7060306@gmail.com> Dear All, I hope you're having a nice day. I coded my program based on the /src/ksp/ksp/example/tutorial/ex45.c. However, I met a problem that the memory is accumulating while the program is running. I have a doubt on the DMDAVecGetArray(da, b, &bb) and DMDAVecRestoreArray(da,b,&bb). Does the former one allocate any memory for 'bb'? If so, does the later one free the memory allocated for the 'bb'? BTW, I just installed the Valgrind. Is there any useful commands in Valgrind can monitor the memory of all the arrays in my program in real time so that I can detect which array keeps requesting the memory? thanks, Alan From bsmith at mcs.anl.gov Thu Aug 30 18:18:05 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Aug 2012 18:18:05 -0500 Subject: [petsc-users] Memory Accumulation In-Reply-To: <503FF27D.7060306@gmail.com> References: <503FF27D.7060306@gmail.com> Message-ID: <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> On Aug 30, 2012, at 6:08 PM, "Zhenglun (Alan) Wei" wrote: > Dear All, > I hope you're having a nice day. > I coded my program based on the /src/ksp/ksp/example/tutorial/ex45.c. However, I met a problem that the memory is accumulating while the program is running. I have a doubt on the DMDAVecGetArray(da, b, &bb) and DMDAVecRestoreArray(da,b,&bb). Does the former one allocate any memory for 'bb'? If so, does the later one free the memory allocated for the 'bb'? You must match each DMDAVecGetArray with a DMDAVecRestoreArray() or you may bleed memory > BTW, I just installed the Valgrind. Is there any useful commands in Valgrind can monitor the memory of all the arrays in my program in real time so that I can detect which array keeps requesting the memory? It is easy to simply using -malloc -malloc_debug -malloc_dump to see all the memory that was allocated and not freed. Barry > > thanks, > Alan From zhenglun.wei at gmail.com Thu Aug 30 18:19:13 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 30 Aug 2012 18:19:13 -0500 Subject: [petsc-users] Memory Accumulation In-Reply-To: <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> References: <503FF27D.7060306@gmail.com> <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> Message-ID: <503FF4F1.1090400@gmail.com> Thanks Barry. It helps a lot. Alan On 8/30/2012 6:18 PM, Barry Smith wrote: > On Aug 30, 2012, at 6:08 PM, "Zhenglun (Alan) Wei" wrote: > >> Dear All, >> I hope you're having a nice day. >> I coded my program based on the /src/ksp/ksp/example/tutorial/ex45.c. However, I met a problem that the memory is accumulating while the program is running. I have a doubt on the DMDAVecGetArray(da, b, &bb) and DMDAVecRestoreArray(da,b,&bb). Does the former one allocate any memory for 'bb'? If so, does the later one free the memory allocated for the 'bb'? > You must match each DMDAVecGetArray with a DMDAVecRestoreArray() or you may bleed memory > >> BTW, I just installed the Valgrind. Is there any useful commands in Valgrind can monitor the memory of all the arrays in my program in real time so that I can detect which array keeps requesting the memory? > It is easy to simply using -malloc -malloc_debug -malloc_dump to see all the memory that was allocated and not freed. > > Barry > >> thanks, >> Alan From Shuangshuang.Jin at pnnl.gov Thu Aug 30 18:20:04 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Thu, 30 Aug 2012 16:20:04 -0700 Subject: [petsc-users] make ex5 Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018F1E823019@EMAIL04.pnl.gov> Hi, there, I got an error while trying to do run an example in PETSc src/snes/examples/tutorials: [d3m956 at olympus tutorials]$ make ex5 *********************W-a-r-n-i-n-g************************* Your PETSC_DIR may not match the directory you are in PETSC_DIR /share/apps/petsc/3.2p3 Current directory /pic/projects/mca/ss/petsc-3.3/src/snes/examples/tutorials Ignore this if you are running make test ****************************************************** mpicc -o ex5.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -I/share/apps/petsc/3.2p3/include -I/share/apps/petsc/3.2p3/gcc-4.1.2-openmpi-1.5.4/includ e -I/share/apps/openmpi/1.5.4/gcc/4.1.2/include -D__INSDIR__=src/snes/examples/tutorials/ ex5.c ex5.c(67): remark #1419: external declaration in primary source file extern PetscErrorCode FormInitialGuess(DM,AppCtx*,Vec); ^ ex5.c(68): remark #1419: external declaration in primary source file extern PetscErrorCode FormFunctionLocal(DMDALocalInfo*,PetscScalar**,PetscScalar**,AppCtx*); ^ ex5.c(69): remark #1419: external declaration in primary source file extern PetscErrorCode FormJacobianLocal(DMDALocalInfo*,PetscScalar**,Mat,AppCtx*); ^ ex5.c(73): remark #1419: external declaration in primary source file extern PetscErrorCode NonlinearGS(SNES,Vec,Vec,void*); ^ ex5.c(107): remark #266: function "SNESSetGS" declared implicitly ierr = SNESSetGS(snes, NonlinearGS, PETSC_NULL);CHKERRQ(ierr); ^ ex5.c(403): remark #869: parameter "ctx" was never referenced PetscErrorCode NonlinearGS(SNES snes,Vec X, Vec B, void * ctx) ^ mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ex5 ex5.o -Wl,-rpath,/share/apps/petsc/3.2p3//gcc-4.1.2-openmpi-1.5.4/lib -L/share/apps/petsc/3.2p 3/gcc-4.1.2-openmpi-1.5.4/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/share/apps/petsc/3.2p3/gcc-4.1.2-openmpi-1.5.4/lib -lflapack -lfblas -Wl,-rpath,/share/apps/openmpi/1.5.4/gcc/4.1.2 /lib -L/share/apps/openmpi/1.5.4/gcc/4.1.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lnsl -lutil -lgcc_s -lpthread -l mpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -ldl -lmpi -lnsl -lutil -lgcc_s -lpthread -ldl /share/apps/intel/11.1/069/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail ex5.o: In function `main': /pic/projects/mca/ss/petsc-3.3/src/snes/examples/tutorials/ex5.c:107: undefined reference to `SNESSetGS' make: [ex5] Error 1 (ignored) /bin/rm -f ex5.o Can anyone tell me what is the problem here? Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 30 18:21:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Aug 2012 18:21:07 -0500 Subject: [petsc-users] mpiaijperm and mpiaijcrl In-Reply-To: References: Message-ID: You have no reason to use those formats. They are specialized for the Cray X1 vector machine which no longer exists. They will not be faster on another machine. Sorry for the confusion. Barry On Aug 30, 2012, at 3:51 PM, Gaetan Kenway wrote: > Hi > > I was wondering if anyone had any experience with using these new matrix formats in PETSc. I have a block aij matrix (with block size 5) and tried converting to either of these types and it just trashed memory for 10 minutes and I then killed it. Matrix assemble is 13 seconds for reference. I then tried to construct the matrix directly with "MatCreateMPIAIJPERM" or "MatCreateMPIAIJCRL". Neither of these functions are wrapped in Fortran. Also the documentation for "MatCreateMPIAIJCRL" is extremely confusing as it says that comm is set to PETSC_COMM_SELF which defeats the purpose of a MPI matrix. > > Therefor, two questions: > 1. Have any users seen a performance benefit of either of these formats (in parallel)? > 2. How does one actually efficiently create such matrices? (from Fortran) > > On a side note, is it possible somehow to indicate on the manual pages all the functions that are not actually wrapped in Fortran? It would make it easier to know which one you can and can't use instead of having the compiler tell you it can't find a symbol. > > Thanks, > Gaetan From bsmith at mcs.anl.gov Thu Aug 30 18:24:47 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Aug 2012 18:24:47 -0500 Subject: [petsc-users] make ex5 In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018F1E823019@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018F1E823019@EMAIL04.pnl.gov> Message-ID: <09986D9A-DC4C-45E5-B6D5-8BAC3B9DAF2E@mcs.anl.gov> On Aug 30, 2012, at 6:20 PM, "Jin, Shuangshuang" wrote: > Hi, there, I got an error while trying to do run an example in PETSc src/snes/examples/tutorials: > > [d3m956 at olympus tutorials]$ make ex5 > *********************W-a-r-n-i-n-g************************* > Your PETSC_DIR may not match the directory you are in > PETSC_DIR /share/apps/petsc/3.2p3 Current directory /pic/projects/mca/ss/petsc-3.3/src/snes/examples/tutorials Looks like you are trying to compile a PETSc 3.3 example with PETSc 3.2. You cannot do this; new stuff was added to 3.3 that is not in 3.2 > ****************************************************** > mpicc -o ex5.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -I/share/apps/petsc/3.2p3/include -I/share/apps/petsc/3.2p3/gcc-4.1.2-openmpi-1.5.4/includ e -I/share/apps/openmpi/1.5.4/gcc/4.1.2/include -D__INSDIR__=src/snes/examples/tutorials/ ex5.c > ex5.c(67): remark #1419: external declaration in primary source file The compiler is being stupid. This is completely legal and legitimate C code; ignore these warnings. Barry > extern PetscErrorCode FormInitialGuess(DM,AppCtx*,Vec); > ^ > > ex5.c(68): remark #1419: external declaration in primary source file > extern PetscErrorCode FormFunctionLocal(DMDALocalInfo*,PetscScalar**,PetscScalar**,AppCtx*); > ^ > > ex5.c(69): remark #1419: external declaration in primary source file > extern PetscErrorCode FormJacobianLocal(DMDALocalInfo*,PetscScalar**,Mat,AppCtx*); > ^ > > ex5.c(73): remark #1419: external declaration in primary source file > extern PetscErrorCode NonlinearGS(SNES,Vec,Vec,void*); > ^ > > ex5.c(107): remark #266: function "SNESSetGS" declared implicitly > ierr = SNESSetGS(snes, NonlinearGS, PETSC_NULL);CHKERRQ(ierr); > ^ > > ex5.c(403): remark #869: parameter "ctx" was never referenced > PetscErrorCode NonlinearGS(SNES snes,Vec X, Vec B, void * ctx) > ^ > > mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ex5 ex5.o -Wl,-rpath,/share/apps/petsc/3.2p3//gcc-4.1.2-openmpi-1.5.4/lib -L/share/apps/petsc/3.2p 3/gcc-4.1.2-openmpi-1.5.4/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/share/apps/petsc/3.2p3/gcc-4.1.2-openmpi-1.5.4/lib -lflapack -lfblas -Wl,-rpath,/share/apps/openmpi/1.5.4/gcc/4.1.2 /lib -L/share/apps/openmpi/1.5.4/gcc/4.1.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lnsl -lutil -lgcc_s -lpthread -l mpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -ldl -lmpi -lnsl -lutil -lgcc_s -lpthread -ldl > /share/apps/intel/11.1/069/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail > ex5.o: In function `main': > /pic/projects/mca/ss/petsc-3.3/src/snes/examples/tutorials/ex5.c:107: undefined reference to `SNESSetGS' > make: [ex5] Error 1 (ignored) > /bin/rm -f ex5.o > > Can anyone tell me what is the problem here? > > Thanks, > Shuangshuang > From jedbrown at mcs.anl.gov Thu Aug 30 18:28:14 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 30 Aug 2012 18:28:14 -0500 Subject: [petsc-users] Memory Accumulation In-Reply-To: <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> References: <503FF27D.7060306@gmail.com> <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> Message-ID: On Thu, Aug 30, 2012 at 6:18 PM, Barry Smith wrote: > It is easy to simply using -malloc -malloc_debug -malloc_dump to see all > the memory that was allocated and not freed. With petsc-3.3 and later, just export PETSC_OPTIONS=-malloc_test. It does the same as above in debug mode, but turns it off for optimized builds so you can just export it in your environment and forget about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 30 18:31:12 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Aug 2012 18:31:12 -0500 Subject: [petsc-users] Memory Accumulation In-Reply-To: References: <503FF27D.7060306@gmail.com> <4972B087-6ABB-4C30-9437-49075BCC2013@mcs.anl.gov> Message-ID: On Aug 30, 2012, at 6:28 PM, Jed Brown wrote: > On Thu, Aug 30, 2012 at 6:18 PM, Barry Smith wrote: > It is easy to simply using -malloc -malloc_debug -malloc_dump to see all the memory that was allocated and not freed. > > With petsc-3.3 and later, just export PETSC_OPTIONS=-malloc_test. It does the same as above in debug mode, but turns it off for optimized builds so you can just export it in your environment and forget about it. Thanks. Added to my .profile and .bashrc Barry From kenway at utias.utoronto.ca Thu Aug 30 18:56:38 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Thu, 30 Aug 2012 19:56:38 -0400 Subject: [petsc-users] mpiaijperm and mpiaijcrl In-Reply-To: References: Message-ID: Good to know. I did actually manage to get them to run after. I had to convert the mpibaij matrix to mpiaij and then to one of those formats. They had almost identically the same performance on a modern intel processor. Thanks for the reply Gaetan On Thu, Aug 30, 2012 at 7:21 PM, Barry Smith wrote: > > You have no reason to use those formats. They are specialized for the > Cray X1 vector machine which no longer exists. They will not be faster on > another machine. Sorry for the confusion. > > Barry > > On Aug 30, 2012, at 3:51 PM, Gaetan Kenway > wrote: > > > Hi > > > > I was wondering if anyone had any experience with using these new matrix > formats in PETSc. I have a block aij matrix (with block size 5) and tried > converting to either of these types and it just trashed memory for 10 > minutes and I then killed it. Matrix assemble is 13 seconds for reference. > I then tried to construct the matrix directly with "MatCreateMPIAIJPERM" > or "MatCreateMPIAIJCRL". Neither of these functions are wrapped in > Fortran. Also the documentation for "MatCreateMPIAIJCRL" is extremely > confusing as it says that comm is set to PETSC_COMM_SELF which defeats the > purpose of a MPI matrix. > > > > Therefor, two questions: > > 1. Have any users seen a performance benefit of either of these formats > (in parallel)? > > 2. How does one actually efficiently create such matrices? (from Fortran) > > > > On a side note, is it possible somehow to indicate on the manual pages > all the functions that are not actually wrapped in Fortran? It would make > it easier to know which one you can and can't use instead of having the > compiler tell you it can't find a symbol. > > > > Thanks, > > Gaetan > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpau at lbl.gov Thu Aug 30 19:24:02 2012 From: gpau at lbl.gov (George Pau) Date: Thu, 30 Aug 2012 17:24:02 -0700 Subject: [petsc-users] pctype hmpi Message-ID: Hi, I have some issues using the -pctype hmpi. I used the same setting found at http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html i.e. -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre_type boomeramg My command is mpiexec -n 1 myprogram But, I get [gilbert:4041] *** An error occurred in MPI_Bcast [gilbert:4041] *** on communicator MPI_COMM_WORLD [gilbert:4041] *** MPI_ERR_COMM: invalid communicator [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) with openmpi. I get similar error with mpich2 Fatal error in PMPI_Bcast: Invalid communicator, error stack: PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed PMPI_Bcast(1418): Invalid communicator I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 30 21:28:47 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Aug 2012 21:28:47 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: Message-ID: On Aug 30, 2012, at 7:24 PM, George Pau wrote: > Hi, > > I have some issues using the -pctype hmpi. I used the same setting found at > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > i.e. > -pc_type hmpi > -ksp_type preonly > -hmpi_ksp_type cg > -hmpi_pc_type hypre > -hmpi_pc_hypre_type boomeramg > > My command is > > mpiexec -n 1 myprogram Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner This will tell you want option to start PETSc up with. I will fix the PC so that it prints a far more useful error message. Barry > > But, I get > > [gilbert:4041] *** An error occurred in MPI_Bcast > [gilbert:4041] *** on communicator MPI_COMM_WORLD > [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > with openmpi. I get similar error with mpich2 > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed > PMPI_Bcast(1418): Invalid communicator > > I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. > > Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > From gpau at lbl.gov Thu Aug 30 22:02:49 2012 From: gpau at lbl.gov (George Pau) Date: Thu, 30 Aug 2012 20:02:49 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: References: Message-ID: Hi Barry, I tried with the addition of -hmpi_spawn_size 3 but I am still getting the same error though. I am using mpich2. Any other options to try? George On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > On Aug 30, 2012, at 7:24 PM, George Pau wrote: > >> Hi, >> >> I have some issues using the -pctype hmpi. I used the same setting found at >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html >> >> i.e. >> -pc_type hmpi >> -ksp_type preonly >> -hmpi_ksp_type cg >> -hmpi_pc_type hypre >> -hmpi_pc_hypre_type boomeramg >> >> My command is >> >> mpiexec -n 1 myprogram > > Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has > > See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner > > This will tell you want option to start PETSc up with. > > I will fix the PC so that it prints a far more useful error message. > > > > Barry > > >> >> But, I get >> >> [gilbert:4041] *** An error occurred in MPI_Bcast >> [gilbert:4041] *** on communicator MPI_COMM_WORLD >> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator >> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) >> >> with openmpi. I get similar error with mpich2 >> >> Fatal error in PMPI_Bcast: Invalid communicator, error stack: >> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed >> PMPI_Bcast(1418): Invalid communicator >> >> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. >> >> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? >> >> -- >> George Pau >> Earth Sciences Division >> Lawrence Berkeley National Laboratory >> One Cyclotron, MS 74-120 >> Berkeley, CA 94720 >> >> (510) 486-7196 >> gpau at lbl.gov >> http://esd.lbl.gov/about/staff/georgepau/ >> > From arun.kamath.m at gmail.com Fri Aug 31 00:38:43 2012 From: arun.kamath.m at gmail.com (Arun Kamath) Date: Fri, 31 Aug 2012 07:38:43 +0200 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options Message-ID: <9152F2AE-60C1-405A-961A-E330512083FF@gmail.com> My idea is to add Petsc options to the control file I have for my code for other options in the code and read them as in an example bit below.. So I was putting in the routines that we would be expecting to use in one file. That seems to be done now. Thanks for all the help. case 'B': control>>numint; switch(numint) { case 5: control>>B5; clear(c,numint); break; case 10: control>>B10; clear(c,numint); break; case 11: control>>B11; clear(c,numint); break; From jedbrown at mcs.anl.gov Fri Aug 31 06:14:01 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 06:14:01 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: <9152F2AE-60C1-405A-961A-E330512083FF@gmail.com> References: <9152F2AE-60C1-405A-961A-E330512083FF@gmail.com> Message-ID: On Fri, Aug 31, 2012 at 12:38 AM, Arun Kamath wrote: > My idea is to add Petsc options to the control file I have for my code for > other options in the code and read them as in an example bit below.. So I > was putting in the routines that we would be expecting to use in one file. > That seems to be done now. Thanks for all the help. > This looks hopelessly cryptic. Why not just pass the name of the options file to PetscInitialize() or PetscOptionsInsertFile(). It's one line and automatically "supports" all current and future options (of which there are infinitely many because options contain a compositional algebra). > > case 'B': control>>numint; > switch(numint) > { > case 5: control>>B5; > clear(c,numint); > break; > case 10: control>>B10; > clear(c,numint); > break; > case 11: control>>B11; > clear(c,numint); > break; > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 31 07:31:06 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 07:31:06 -0500 Subject: [petsc-users] Hypre through Petsc: BoomerAMG options In-Reply-To: References: <9152F2AE-60C1-405A-961A-E330512083FF@gmail.com> Message-ID: <2634B0B6-9CEB-45B6-9116-CAD0FD634A8C@mcs.anl.gov> On Aug 31, 2012, at 6:14 AM, Jed Brown wrote: > On Fri, Aug 31, 2012 at 12:38 AM, Arun Kamath wrote: > My idea is to add Petsc options to the control file I have for my code for other options in the code and read them as in an example bit below.. So I was putting in the routines that we would be expecting to use in one file. That seems to be done now. Thanks for all the help. > > This looks hopelessly cryptic. Why not just pass the name of the options file to PetscInitialize() or PetscOptionsInsertFile(). It's one line and automatically "supports" all current and future options (of which there are infinitely many because options contain a compositional algebra). > > > case 'B': control>>numint; > switch(numint) > { > case 5: control>>B5; > clear(c,numint); > break; > case 10: control>>B10; > clear(c,numint); > break; > case 11: control>>B11; > clear(c,numint); > break; > Note that it is also possible to include your options and PETSc options in the same file. So for example you may have a file with something like the following # My options # myoption 1 myoption 2 etc # # PETSc options # -ksp_type fgmres -pc_type hypre etc just make sure your options processing skips or ignores the PETSc options. PETSc will ignore your options in reading the file. Barry From bsmith at mcs.anl.gov Fri Aug 31 11:30:12 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 11:30:12 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: Message-ID: <97AC8886-6AD0-46A8-B301-F1A085F49C64@mcs.anl.gov> George, I am having trouble running this myself. I am debugging now and will get back to you. Barry On Aug 30, 2012, at 10:02 PM, George Pau wrote: > Hi Barry, > > I tried with the addition of > > -hmpi_spawn_size 3 > > but I am still getting the same error though. I am using mpich2. Any other options to try? > > George > > > On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > >> >> On Aug 30, 2012, at 7:24 PM, George Pau wrote: >> >>> Hi, >>> >>> I have some issues using the -pctype hmpi. I used the same setting found at >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html >>> >>> i.e. >>> -pc_type hmpi >>> -ksp_type preonly >>> -hmpi_ksp_type cg >>> -hmpi_pc_type hypre >>> -hmpi_pc_hypre_type boomeramg >>> >>> My command is >>> >>> mpiexec -n 1 myprogram >> >> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has >> >> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner >> >> This will tell you want option to start PETSc up with. >> >> I will fix the PC so that it prints a far more useful error message. >> >> >> >> Barry >> >> >>> >>> But, I get >>> >>> [gilbert:4041] *** An error occurred in MPI_Bcast >>> [gilbert:4041] *** on communicator MPI_COMM_WORLD >>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator >>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) >>> >>> with openmpi. I get similar error with mpich2 >>> >>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: >>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed >>> PMPI_Bcast(1418): Invalid communicator >>> >>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. >>> >>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? >>> >>> -- >>> George Pau >>> Earth Sciences Division >>> Lawrence Berkeley National Laboratory >>> One Cyclotron, MS 74-120 >>> Berkeley, CA 94720 >>> >>> (510) 486-7196 >>> gpau at lbl.gov >>> http://esd.lbl.gov/about/staff/georgepau/ >>> >> > From bsmith at mcs.anl.gov Fri Aug 31 13:17:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 13:17:07 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: Message-ID: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > Hi Barry, > > I tried with the addition of > > -hmpi_spawn_size 3 > > but I am still getting the same error though. The EXACT same error? Or some other error? What happens if you run with the -hmpi_merge_size option instead? Barry 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > I am using mpich2. Any other options to try? > > George > > > On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > >> >> On Aug 30, 2012, at 7:24 PM, George Pau wrote: >> >>> Hi, >>> >>> I have some issues using the -pctype hmpi. I used the same setting found at >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html >>> >>> i.e. >>> -pc_type hmpi >>> -ksp_type preonly >>> -hmpi_ksp_type cg >>> -hmpi_pc_type hypre >>> -hmpi_pc_hypre_type boomeramg >>> >>> My command is >>> >>> mpiexec -n 1 myprogram >> >> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has >> >> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner >> >> This will tell you want option to start PETSc up with. >> >> I will fix the PC so that it prints a far more useful error message. >> >> >> >> Barry >> >> >>> >>> But, I get >>> >>> [gilbert:4041] *** An error occurred in MPI_Bcast >>> [gilbert:4041] *** on communicator MPI_COMM_WORLD >>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator >>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) >>> >>> with openmpi. I get similar error with mpich2 >>> >>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: >>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed >>> PMPI_Bcast(1418): Invalid communicator >>> >>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. >>> >>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? >>> >>> -- >>> George Pau >>> Earth Sciences Division >>> Lawrence Berkeley National Laboratory >>> One Cyclotron, MS 74-120 >>> Berkeley, CA 94720 >>> >>> (510) 486-7196 >>> gpau at lbl.gov >>> http://esd.lbl.gov/about/staff/georgepau/ >>> >> > From gpau at lbl.gov Fri Aug 31 13:27:31 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 11:27:31 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> Message-ID: <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> Hi Barry, 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. George On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > On Aug 30, 2012, at 10:02 PM, George Pau wrote: > >> Hi Barry, >> >> I tried with the addition of >> >> -hmpi_spawn_size 3 >> >> but I am still getting the same error though. > > The EXACT same error? Or some other error? > > What happens if you run with the -hmpi_merge_size option instead? > > Barry > > 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. > > 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > > >> I am using mpich2. Any other options to try? >> >> George >> >> >> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: >> >>> >>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: >>> >>>> Hi, >>>> >>>> I have some issues using the -pctype hmpi. I used the same setting found at >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html >>>> >>>> i.e. >>>> -pc_type hmpi >>>> -ksp_type preonly >>>> -hmpi_ksp_type cg >>>> -hmpi_pc_type hypre >>>> -hmpi_pc_hypre_type boomeramg >>>> >>>> My command is >>>> >>>> mpiexec -n 1 myprogram >>> >>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has >>> >>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner >>> >>> This will tell you want option to start PETSc up with. >>> >>> I will fix the PC so that it prints a far more useful error message. >>> >>> >>> >>> Barry >>> >>> >>>> >>>> But, I get >>>> >>>> [gilbert:4041] *** An error occurred in MPI_Bcast >>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD >>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator >>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) >>>> >>>> with openmpi. I get similar error with mpich2 >>>> >>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: >>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed >>>> PMPI_Bcast(1418): Invalid communicator >>>> >>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. >>>> >>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? >>>> >>>> -- >>>> George Pau >>>> Earth Sciences Division >>>> Lawrence Berkeley National Laboratory >>>> One Cyclotron, MS 74-120 >>>> Berkeley, CA 94720 >>>> >>>> (510) 486-7196 >>>> gpau at lbl.gov >>>> http://esd.lbl.gov/about/staff/georgepau/ >>>> >>> >> > From bsmith at mcs.anl.gov Fri Aug 31 13:31:33 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 13:31:33 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> Message-ID: On Aug 31, 2012, at 1:27 PM, George Pau wrote: > Hi Barry, > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. That should not happen. Please run and send all the output including the exact command line used > I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 and send the complete error message. Barry > > George > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > >> >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: >> >>> Hi Barry, >>> >>> I tried with the addition of >>> >>> -hmpi_spawn_size 3 >>> >>> but I am still getting the same error though. >> >> The EXACT same error? Or some other error? >> >> What happens if you run with the -hmpi_merge_size option instead? >> >> Barry >> >> 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. >> >> 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. >> >> >>> I am using mpich2. Any other options to try? >>> >>> George >>> >>> >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: >>> >>>> >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: >>>> >>>>> Hi, >>>>> >>>>> I have some issues using the -pctype hmpi. I used the same setting found at >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html >>>>> >>>>> i.e. >>>>> -pc_type hmpi >>>>> -ksp_type preonly >>>>> -hmpi_ksp_type cg >>>>> -hmpi_pc_type hypre >>>>> -hmpi_pc_hypre_type boomeramg >>>>> >>>>> My command is >>>>> >>>>> mpiexec -n 1 myprogram >>>> >>>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has >>>> >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner >>>> >>>> This will tell you want option to start PETSc up with. >>>> >>>> I will fix the PC so that it prints a far more useful error message. >>>> >>>> >>>> >>>> Barry >>>> >>>> >>>>> >>>>> But, I get >>>>> >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) >>>>> >>>>> with openmpi. I get similar error with mpich2 >>>>> >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed >>>>> PMPI_Bcast(1418): Invalid communicator >>>>> >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. >>>>> >>>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? >>>>> >>>>> -- >>>>> George Pau >>>>> Earth Sciences Division >>>>> Lawrence Berkeley National Laboratory >>>>> One Cyclotron, MS 74-120 >>>>> Berkeley, CA 94720 >>>>> >>>>> (510) 486-7196 >>>>> gpau at lbl.gov >>>>> http://esd.lbl.gov/about/staff/georgepau/ >>>>> >>>> >>> >> > From gpau at lbl.gov Fri Aug 31 15:09:58 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 13:09:58 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> Message-ID: Hi Barry, For the hmpi_spawn_size, the options in my .petscrc are -info -pc_view pc_type hmpi -ksp_type preonly -ksp_view -hmpi_pc_monitor -hmpi_ksp_monitor -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre_type boomeramg -hmpi_spawn_size 3 mpiexec.mpich2 -n 1 myprogram [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [0] petscinitialize_(): Running on machine: gilbert [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 [0] MatSetUp(): Warning not preallocating matrix storage [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 Fatal error in PMPI_Bcast: Invalid communicator, error stack: PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, root=0, comm=0x0) failed PMPI_Bcast(1418): Invalid communicator I inserted some print statement between the ksp calls and found that the error occurs in call KSPSetFromOptions(ksp, pierr) 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job by mpiexec.mpich2 -n 2 myprogram [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [0] petscinitialize_(): Running on machine: gilbert [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [1] petscinitialize_(): Running on machine: gilbert [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 2147483647 [0] MatSetUp(): Warning not preallocating matrix storage [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 2147483647 [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 2147483647 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] MatStashScatterBegin_Private(): No of messages: 1 [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter [0] VecScatterCreate(): General case: MPI to Seq [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] PCSetUp(): Setting up new PC [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCCreate_HMPI() line 283 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c [0]PETSC ERROR: PCSetType() line 83 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c [0]PETSC ERROR: PCSetFromOptions() line 188 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c [0]PETSC ERROR: KSPSetFromOptions() line 287 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: PC does not have apply! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCApply() line 382 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPInitialResidual() line 64 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: KSPSolve_GMRES() line 230 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 446 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c I note that the error appears to occur at the same point. George On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith wrote: > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > Hi Barry, > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > That should not happen. Please run and send all the output including > the exact command line used > > > > I am currently using the MPICH2 distribution provided by ubuntu but if > MPICH version that Petsc download with -download-mpich works, I can use > that. > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more > than 1 cpus. But, petsc will complain that the pctype hmpi can only be > used in serial. > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 > and send the complete error message. > > > Barry > > > > > George > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > >> > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > >> > >>> Hi Barry, > >>> > >>> I tried with the addition of > >>> > >>> -hmpi_spawn_size 3 > >>> > >>> but I am still getting the same error though. > >> > >> The EXACT same error? Or some other error? > >> > >> What happens if you run with the -hmpi_merge_size option > instead? > >> > >> Barry > >> > >> 1) I am getting a crash with the spawn version I suspect is due to bugs > in the MPICH version I am using related to spawn. > >> > >> 2) I am getting errors with the merge version due to Apple's ASLR which > they make hard to turn off. > >> > >> > >>> I am using mpich2. Any other options to try? > >>> > >>> George > >>> > >>> > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > >>> > >>>> > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I have some issues using the -pctype hmpi. I used the same setting > found at > >>>>> > >>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > >>>>> > >>>>> i.e. > >>>>> -pc_type hmpi > >>>>> -ksp_type preonly > >>>>> -hmpi_ksp_type cg > >>>>> -hmpi_pc_type hypre > >>>>> -hmpi_pc_hypre_type boomeramg > >>>>> > >>>>> My command is > >>>>> > >>>>> mpiexec -n 1 myprogram > >>>> > >>>> Sorry the documentation doesn't make this clearer. You need to start > PETSc with special options to get the "worker" processes initialized. From > the manual page for PCHMPI it has > >>>> > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up > MPI for use with this preconditioner > >>>> > >>>> This will tell you want option to start PETSc up with. > >>>> > >>>> I will fix the PC so that it prints a far more useful error message. > >>>> > >>>> > >>>> > >>>> Barry > >>>> > >>>> > >>>>> > >>>>> But, I get > >>>>> > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > >>>>> > >>>>> with openmpi. I get similar error with mpich2 > >>>>> > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, > root=0, comm=0x0) failed > >>>>> PMPI_Bcast(1418): Invalid communicator > >>>>> > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 > and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi > --download-hypre=1 and I am on a Ubuntu machine. > >>>>> > >>>>> Note that with the default pc_type and ksp_type, everything is fine. > It was also tested with multiple processors. I wondering whether there > are some options that I am not specifying correctly? > >>>>> > >>>>> -- > >>>>> George Pau > >>>>> Earth Sciences Division > >>>>> Lawrence Berkeley National Laboratory > >>>>> One Cyclotron, MS 74-120 > >>>>> Berkeley, CA 94720 > >>>>> > >>>>> (510) 486-7196 > >>>>> gpau at lbl.gov > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > >>>>> > >>>> > >>> > >> > > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 31 15:18:54 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 15:18:54 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> Message-ID: <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> On Aug 31, 2012, at 3:09 PM, George Pau wrote: > Hi Barry, > > For the hmpi_spawn_size, the options in my .petscrc are > -info > -pc_view > pc_type hmpi How come there is no - in front of this one? > -ksp_type preonly > -ksp_view > -hmpi_pc_monitor > -hmpi_ksp_monitor > -hmpi_ksp_type cg > -hmpi_pc_type hypre > -hmpi_pc_hypre_type boomeramg > -hmpi_spawn_size 3 > > mpiexec.mpich2 -n 1 myprogram > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [0] petscinitialize_(): Running on machine: gilbert > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 > [0] MatSetUp(): Warning not preallocating matrix storage > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, root=0, comm=0x0) failed > PMPI_Bcast(1418): Invalid communicator > > I inserted some print statement between the ksp calls and found that the error occurs in > > call KSPSetFromOptions(ksp, pierr) > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job by How come there is no - in front of hmpi_merge_size 2? Can you try putting all the arguments as command line arguments instead of in a file? It shouldn't matter but it seems like some of the arguments are being ignored. Barry > > mpiexec.mpich2 -n 2 myprogram > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [0] petscinitialize_(): Running on machine: gilbert > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [1] petscinitialize_(): Running on machine: gilbert > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 2147483647 > [0] MatSetUp(): Warning not preallocating matrix storage > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 2147483647 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > [0] MatStashScatterBegin_Private(): No of messages: 1 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCCreate_HMPI() line 283 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > [0]PETSC ERROR: PCSetType() line 83 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: PCSetFromOptions() line 188 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: KSPSetFromOptions() line 287 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: PC does not have apply! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApply() line 382 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPInitialResidual() line 64 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 446 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > I note that the error appears to occur at the same point. > > George > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith wrote: > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > Hi Barry, > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > That should not happen. Please run and send all the output including the exact command line used > > > > I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 and send the complete error message. > > > Barry > > > > > George > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > >> > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > >> > >>> Hi Barry, > >>> > >>> I tried with the addition of > >>> > >>> -hmpi_spawn_size 3 > >>> > >>> but I am still getting the same error though. > >> > >> The EXACT same error? Or some other error? > >> > >> What happens if you run with the -hmpi_merge_size option instead? > >> > >> Barry > >> > >> 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. > >> > >> 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > >> > >> > >>> I am using mpich2. Any other options to try? > >>> > >>> George > >>> > >>> > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > >>> > >>>> > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I have some issues using the -pctype hmpi. I used the same setting found at > >>>>> > >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > >>>>> > >>>>> i.e. > >>>>> -pc_type hmpi > >>>>> -ksp_type preonly > >>>>> -hmpi_ksp_type cg > >>>>> -hmpi_pc_type hypre > >>>>> -hmpi_pc_hypre_type boomeramg > >>>>> > >>>>> My command is > >>>>> > >>>>> mpiexec -n 1 myprogram > >>>> > >>>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has > >>>> > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner > >>>> > >>>> This will tell you want option to start PETSc up with. > >>>> > >>>> I will fix the PC so that it prints a far more useful error message. > >>>> > >>>> > >>>> > >>>> Barry > >>>> > >>>> > >>>>> > >>>>> But, I get > >>>>> > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > >>>>> > >>>>> with openmpi. I get similar error with mpich2 > >>>>> > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed > >>>>> PMPI_Bcast(1418): Invalid communicator > >>>>> > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. > >>>>> > >>>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? > >>>>> > >>>>> -- > >>>>> George Pau > >>>>> Earth Sciences Division > >>>>> Lawrence Berkeley National Laboratory > >>>>> One Cyclotron, MS 74-120 > >>>>> Berkeley, CA 94720 > >>>>> > >>>>> (510) 486-7196 > >>>>> gpau at lbl.gov > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > >>>>> > >>>> > >>> > >> > > > > > > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > From gpau at lbl.gov Fri Aug 31 15:30:35 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 13:30:35 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> Message-ID: Sorry, it was a cut and paste error. I tried running the code with all the options in the command line: mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg but I get the exact same outputs. George On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith wrote: > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > Hi Barry, > > > > For the hmpi_spawn_size, the options in my .petscrc are > > -info > > -pc_view > > pc_type hmpi > > How come there is no - in front of this one? > > > -ksp_type preonly > > -ksp_view > > -hmpi_pc_monitor > > -hmpi_ksp_monitor > > -hmpi_ksp_type cg > > -hmpi_pc_type hypre > > -hmpi_pc_hypre_type boomeramg > > -hmpi_spawn_size 3 > > > > mpiexec.mpich2 -n 1 myprogram > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > > [0] MatSetUp(): Warning not preallocating matrix storage > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 > unneeded,3222 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is > 360 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode > routines > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, > root=0, comm=0x0) failed > > PMPI_Bcast(1418): Invalid communicator > > > > I inserted some print statement between the ksp calls and found that the > error occurs in > > > > call KSPSetFromOptions(ksp, pierr) > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job > by > > How come there is no - in front of hmpi_merge_size 2? > > > Can you try putting all the arguments as command line arguments instead > of in a file? It shouldn't matter but it seems like some of the arguments > are being ignored. > > Barry > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [1] petscinitialize_(): Running on machine: gilbert > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374780 max tags = 2147483647 > > [0] MatSetUp(): Warning not preallocating matrix storage > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374782 max tags = 2147483647 > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374777 max tags = 2147483647 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374780 max tags = 2147483647 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 > unneeded,1602 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is > 180 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode > routines > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 > unneeded,1602 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is > 180 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode > routines > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > [0] VecScatterCreate(): General case: MPI to Seq > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 > unneeded,9 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374782 > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 > unneeded,9 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374782 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > [0] PCSetUp(): Setting up new PC > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Nonconforming object sizes! > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 13:00:31 2012 > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > [0]PETSC ERROR: PCSetType() line 83 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > [0]PETSC ERROR: PCSetFromOptions() line 188 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: No support for this operation for this object type! > > [0]PETSC ERROR: PC does not have apply! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 13:00:31 2012 > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PCApply() line 382 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: KSPInitialResidual() line 64 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve() line 446 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > I note that the error appears to occur at the same point. > > > > George > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith > wrote: > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > Hi Barry, > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > > > That should not happen. Please run and send all the output including > the exact command line used > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu but > if MPICH version that Petsc download with -download-mpich works, I can use > that. > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more > than 1 cpus. But, petsc will complain that the pctype hmpi can only be > used in serial. > > > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size > 2 and send the complete error message. > > > > > > Barry > > > > > > > > George > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > >> > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > >> > > >>> Hi Barry, > > >>> > > >>> I tried with the addition of > > >>> > > >>> -hmpi_spawn_size 3 > > >>> > > >>> but I am still getting the same error though. > > >> > > >> The EXACT same error? Or some other error? > > >> > > >> What happens if you run with the -hmpi_merge_size option > instead? > > >> > > >> Barry > > >> > > >> 1) I am getting a crash with the spawn version I suspect is due to > bugs in the MPICH version I am using related to spawn. > > >> > > >> 2) I am getting errors with the merge version due to Apple's ASLR > which they make hard to turn off. > > >> > > >> > > >>> I am using mpich2. Any other options to try? > > >>> > > >>> George > > >>> > > >>> > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > >>> > > >>>> > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > >>>> > > >>>>> Hi, > > >>>>> > > >>>>> I have some issues using the -pctype hmpi. I used the same > setting found at > > >>>>> > > >>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > >>>>> > > >>>>> i.e. > > >>>>> -pc_type hmpi > > >>>>> -ksp_type preonly > > >>>>> -hmpi_ksp_type cg > > >>>>> -hmpi_pc_type hypre > > >>>>> -hmpi_pc_hypre_type boomeramg > > >>>>> > > >>>>> My command is > > >>>>> > > >>>>> mpiexec -n 1 myprogram > > >>>> > > >>>> Sorry the documentation doesn't make this clearer. You need to > start PETSc with special options to get the "worker" processes initialized. > From the manual page for PCHMPI it has > > >>>> > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up > MPI for use with this preconditioner > > >>>> > > >>>> This will tell you want option to start PETSc up with. > > >>>> > > >>>> I will fix the PC so that it prints a far more useful error message. > > >>>> > > >>>> > > >>>> > > >>>> Barry > > >>>> > > >>>> > > >>>>> > > >>>>> But, I get > > >>>>> > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now > abort) > > >>>>> > > >>>>> with openmpi. I get similar error with mpich2 > > >>>>> > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, > root=0, comm=0x0) failed > > >>>>> PMPI_Bcast(1418): Invalid communicator > > >>>>> > > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 > and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi > --download-hypre=1 and I am on a Ubuntu machine. > > >>>>> > > >>>>> Note that with the default pc_type and ksp_type, everything is > fine. It was also tested with multiple processors. I wondering whether > there are some options that I am not specifying correctly? > > >>>>> > > >>>>> -- > > >>>>> George Pau > > >>>>> Earth Sciences Division > > >>>>> Lawrence Berkeley National Laboratory > > >>>>> One Cyclotron, MS 74-120 > > >>>>> Berkeley, CA 94720 > > >>>>> > > >>>>> (510) 486-7196 > > >>>>> gpau at lbl.gov > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > >>>>> > > >>>> > > >>> > > >> > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 31 16:04:55 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 16:04:55 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> Message-ID: Yikes. It is totally my fault. The handling of these merge and spawn options is done only for PetscInitialize() for C. Not for Fortran, hence the arguments just got ignored. Please find attached a file zstart.c put it in the directory src/sys/ftn-custom and run make in that directory (with appropriate PETSC_DIR and PETSC_ARCH set) Then link and run the example again. Barry On Aug 31, 2012, at 3:30 PM, George Pau wrote: > Sorry, it was a cut and paste error. I tried running the code with all the options in the command line: > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > but I get the exact same outputs. > > George > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith wrote: > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > Hi Barry, > > > > For the hmpi_spawn_size, the options in my .petscrc are > > -info > > -pc_view > > pc_type hmpi > > How come there is no - in front of this one? > > > -ksp_type preonly > > -ksp_view > > -hmpi_pc_monitor > > -hmpi_ksp_monitor > > -hmpi_ksp_type cg > > -hmpi_pc_type hypre > > -hmpi_pc_hypre_type boomeramg > > -hmpi_spawn_size 3 > > > > mpiexec.mpich2 -n 1 myprogram > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 > > [0] MatSetUp(): Warning not preallocating matrix storage > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, root=0, comm=0x0) failed > > PMPI_Bcast(1418): Invalid communicator > > > > I inserted some print statement between the ksp calls and found that the error occurs in > > > > call KSPSetFromOptions(ksp, pierr) > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job by > > How come there is no - in front of hmpi_merge_size 2? > > > Can you try putting all the arguments as command line arguments instead of in a file? It shouldn't matter but it seems like some of the arguments are being ignored. > > Barry > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > [1] petscinitialize_(): Running on machine: gilbert > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 2147483647 > > [0] MatSetUp(): Warning not preallocating matrix storage > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 2147483647 > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 2147483647 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > [0] VecScatterCreate(): General case: MPI to Seq > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > [0] PCSetUp(): Setting up new PC > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: Nonconforming object sizes! > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > [0]PETSC ERROR: PCSetType() line 83 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > [0]PETSC ERROR: PCSetFromOptions() line 188 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: No support for this operation for this object type! > > [0]PETSC ERROR: PC does not have apply! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: PCApply() line 382 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: KSPInitialResidual() line 64 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve() line 446 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > I note that the error appears to occur at the same point. > > > > George > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith wrote: > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > Hi Barry, > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > > > That should not happen. Please run and send all the output including the exact command line used > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. > > > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 and send the complete error message. > > > > > > Barry > > > > > > > > George > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > >> > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > >> > > >>> Hi Barry, > > >>> > > >>> I tried with the addition of > > >>> > > >>> -hmpi_spawn_size 3 > > >>> > > >>> but I am still getting the same error though. > > >> > > >> The EXACT same error? Or some other error? > > >> > > >> What happens if you run with the -hmpi_merge_size option instead? > > >> > > >> Barry > > >> > > >> 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. > > >> > > >> 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > > >> > > >> > > >>> I am using mpich2. Any other options to try? > > >>> > > >>> George > > >>> > > >>> > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > >>> > > >>>> > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > >>>> > > >>>>> Hi, > > >>>>> > > >>>>> I have some issues using the -pctype hmpi. I used the same setting found at > > >>>>> > > >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > >>>>> > > >>>>> i.e. > > >>>>> -pc_type hmpi > > >>>>> -ksp_type preonly > > >>>>> -hmpi_ksp_type cg > > >>>>> -hmpi_pc_type hypre > > >>>>> -hmpi_pc_hypre_type boomeramg > > >>>>> > > >>>>> My command is > > >>>>> > > >>>>> mpiexec -n 1 myprogram > > >>>> > > >>>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has > > >>>> > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner > > >>>> > > >>>> This will tell you want option to start PETSc up with. > > >>>> > > >>>> I will fix the PC so that it prints a far more useful error message. > > >>>> > > >>>> > > >>>> > > >>>> Barry > > >>>> > > >>>> > > >>>>> > > >>>>> But, I get > > >>>>> > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > >>>>> > > >>>>> with openmpi. I get similar error with mpich2 > > >>>>> > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed > > >>>>> PMPI_Bcast(1418): Invalid communicator > > >>>>> > > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. > > >>>>> > > >>>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? > > >>>>> > > >>>>> -- > > >>>>> George Pau > > >>>>> Earth Sciences Division > > >>>>> Lawrence Berkeley National Laboratory > > >>>>> One Cyclotron, MS 74-120 > > >>>>> Berkeley, CA 94720 > > >>>>> > > >>>>> (510) 486-7196 > > >>>>> gpau at lbl.gov > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > >>>>> > > >>>> > > >>> > > >> > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > From gpau at lbl.gov Fri Aug 31 16:07:11 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 14:07:11 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> Message-ID: Hi Barry, You forgot the file ... George On Fri, Aug 31, 2012 at 2:04 PM, Barry Smith wrote: > > Yikes. It is totally my fault. The handling of these merge and spawn > options is done only for PetscInitialize() for C. Not for Fortran, hence > the arguments just got ignored. > > Please find attached a file zstart.c put it in the directory > src/sys/ftn-custom and run make in that directory (with appropriate > PETSC_DIR and PETSC_ARCH set) > > Then link and run the example again. > > > Barry > > > On Aug 31, 2012, at 3:30 PM, George Pau wrote: > > > Sorry, it was a cut and paste error. I tried running the code with all > the options in the command line: > > > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi -ksp_type > preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi -ksp_type > preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > but I get the exact same outputs. > > > > George > > > > > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith wrote: > > > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > > > Hi Barry, > > > > > > For the hmpi_spawn_size, the options in my .petscrc are > > > -info > > > -pc_view > > > pc_type hmpi > > > > How come there is no - in front of this one? > > > > > -ksp_type preonly > > > -ksp_view > > > -hmpi_pc_monitor > > > -hmpi_ksp_monitor > > > -hmpi_ksp_type cg > > > -hmpi_pc_type hypre > > > -hmpi_pc_hypre_type boomeramg > > > -hmpi_spawn_size 3 > > > > > > mpiexec.mpich2 -n 1 myprogram > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: > 3978 unneeded,3222 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 360 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using > Inode routines > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > > > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, > root=0, comm=0x0) failed > > > PMPI_Bcast(1418): Invalid communicator > > > > > > I inserted some print statement between the ksp calls and found that > the error occurs in > > > > > > call KSPSetFromOptions(ksp, pierr) > > > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my > job by > > > > How come there is no - in front of hmpi_merge_size 2? > > > > > > Can you try putting all the arguments as command line arguments > instead of in a file? It shouldn't matter but it seems like some of the > arguments are being ignored. > > > > Barry > > > > > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > [0] petscinitialize_(): Running on machine: gilbert > > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > [1] petscinitialize_(): Running on machine: gilbert > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374780 max tags = 2147483647 > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374782 max tags = 2147483647 > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374777 max tags = 2147483647 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374780 max tags = 2147483647 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: > 1998 unneeded,1602 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 180 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: > 1998 unneeded,1602 used > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 180 > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374780 > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > > [0] VecScatterCreate(): General case: MPI to Seq > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 > unneeded,9 used > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 3 > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374782 > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 > unneeded,9 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 3 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374782 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > > [0] PCSetUp(): Setting up new PC > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > > > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > [0]PETSC ERROR: Nonconforming object sizes! > > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 13:00:31 2012 > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > > [0]PETSC ERROR: PCSetType() line 83 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > [0]PETSC ERROR: PCSetFromOptions() line 188 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > [0]PETSC ERROR: No support for this operation for this object type! > > > [0]PETSC ERROR: PC does not have apply! > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 13:00:31 2012 > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: PCApply() line 382 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > > [0]PETSC ERROR: KSPInitialResidual() line 64 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > > [0]PETSC ERROR: KSPSolve() line 446 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > > > I note that the error appears to occur at the same point. > > > > > > George > > > > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith > wrote: > > > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > > > Hi Barry, > > > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > > > > > That should not happen. Please run and send all the output > including the exact command line used > > > > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu but > if MPICH version that Petsc download with -download-mpich works, I can use > that. > > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more > than 1 cpus. But, petsc will complain that the pctype hmpi can only be > used in serial. > > > > > > That should not happen. Run with 2 MPI processes and > -hmpi_merge_size 2 and send the complete error message. > > > > > > > > > Barry > > > > > > > > > > > George > > > > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > > > >> > > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > > >> > > > >>> Hi Barry, > > > >>> > > > >>> I tried with the addition of > > > >>> > > > >>> -hmpi_spawn_size 3 > > > >>> > > > >>> but I am still getting the same error though. > > > >> > > > >> The EXACT same error? Or some other error? > > > >> > > > >> What happens if you run with the -hmpi_merge_size option > instead? > > > >> > > > >> Barry > > > >> > > > >> 1) I am getting a crash with the spawn version I suspect is due to > bugs in the MPICH version I am using related to spawn. > > > >> > > > >> 2) I am getting errors with the merge version due to Apple's ASLR > which they make hard to turn off. > > > >> > > > >> > > > >>> I am using mpich2. Any other options to try? > > > >>> > > > >>> George > > > >>> > > > >>> > > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > > >>> > > > >>>> > > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > > >>>> > > > >>>>> Hi, > > > >>>>> > > > >>>>> I have some issues using the -pctype hmpi. I used the same > setting found at > > > >>>>> > > > >>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > > >>>>> > > > >>>>> i.e. > > > >>>>> -pc_type hmpi > > > >>>>> -ksp_type preonly > > > >>>>> -hmpi_ksp_type cg > > > >>>>> -hmpi_pc_type hypre > > > >>>>> -hmpi_pc_hypre_type boomeramg > > > >>>>> > > > >>>>> My command is > > > >>>>> > > > >>>>> mpiexec -n 1 myprogram > > > >>>> > > > >>>> Sorry the documentation doesn't make this clearer. You need to > start PETSc with special options to get the "worker" processes initialized. > From the manual page for PCHMPI it has > > > >>>> > > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start > up MPI for use with this preconditioner > > > >>>> > > > >>>> This will tell you want option to start PETSc up with. > > > >>>> > > > >>>> I will fix the PC so that it prints a far more useful error > message. > > > >>>> > > > >>>> > > > >>>> > > > >>>> Barry > > > >>>> > > > >>>> > > > >>>>> > > > >>>>> But, I get > > > >>>>> > > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now > abort) > > > >>>>> > > > >>>>> with openmpi. I get similar error with mpich2 > > > >>>>> > > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, > MPI_INT, root=0, comm=0x0) failed > > > >>>>> PMPI_Bcast(1418): Invalid communicator > > > >>>>> > > > >>>>> I couldn't figure out what is wrong. My petsc is version > 3.3.3 and the configuration is -with-debugging=0 > --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu > machine. > > > >>>>> > > > >>>>> Note that with the default pc_type and ksp_type, everything is > fine. It was also tested with multiple processors. I wondering whether > there are some options that I am not specifying correctly? > > > >>>>> > > > >>>>> -- > > > >>>>> George Pau > > > >>>>> Earth Sciences Division > > > >>>>> Lawrence Berkeley National Laboratory > > > >>>>> One Cyclotron, MS 74-120 > > > >>>>> Berkeley, CA 94720 > > > >>>>> > > > >>>>> (510) 486-7196 > > > >>>>> gpau at lbl.gov > > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > -- > > > George Pau > > > Earth Sciences Division > > > Lawrence Berkeley National Laboratory > > > One Cyclotron, MS 74-120 > > > Berkeley, CA 94720 > > > > > > (510) 486-7196 > > > gpau at lbl.gov > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 31 16:09:54 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 16:09:54 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> Message-ID: <2EABC8B5-DF3A-4B05-A49E-E26566C31979@mcs.anl.gov> [see attached file: zstart.c] On Aug 31, 2012, at 4:07 PM, George Pau wrote: > Hi Barry, > > You forgot the file ... > > George > > On Fri, Aug 31, 2012 at 2:04 PM, Barry Smith wrote: > > Yikes. It is totally my fault. The handling of these merge and spawn options is done only for PetscInitialize() for C. Not for Fortran, hence the arguments just got ignored. > > Please find attached a file zstart.c put it in the directory src/sys/ftn-custom and run make in that directory (with appropriate PETSC_DIR and PETSC_ARCH set) > > Then link and run the example again. > > > Barry > > > On Aug 31, 2012, at 3:30 PM, George Pau wrote: > > > Sorry, it was a cut and paste error. I tried running the code with all the options in the command line: > > > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > but I get the exact same outputs. > > > > George > > > > > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith wrote: > > > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > > > Hi Barry, > > > > > > For the hmpi_spawn_size, the options in my .petscrc are > > > -info > > > -pc_view > > > pc_type hmpi > > > > How come there is no - in front of this one? > > > > > -ksp_type preonly > > > -ksp_view > > > -hmpi_pc_monitor > > > -hmpi_ksp_monitor > > > -hmpi_ksp_type cg > > > -hmpi_pc_type hypre > > > -hmpi_pc_hypre_type boomeramg > > > -hmpi_spawn_size 3 > > > > > > mpiexec.mpich2 -n 1 myprogram > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, root=0, comm=0x0) failed > > > PMPI_Bcast(1418): Invalid communicator > > > > > > I inserted some print statement between the ksp calls and found that the error occurs in > > > > > > call KSPSetFromOptions(ksp, pierr) > > > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job by > > > > How come there is no - in front of hmpi_merge_size 2? > > > > > > Can you try putting all the arguments as command line arguments instead of in a file? It shouldn't matter but it seems like some of the arguments are being ignored. > > > > Barry > > > > > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > [0] petscinitialize_(): Running on machine: gilbert > > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > [1] petscinitialize_(): Running on machine: gilbert > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 2147483647 > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 2147483647 > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 2147483647 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > > [0] VecScatterCreate(): General case: MPI to Seq > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > [0] PCSetUp(): Setting up new PC > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > > [0]PETSC ERROR: Nonconforming object sizes! > > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > > [0]PETSC ERROR: PCSetType() line 83 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > [0]PETSC ERROR: PCSetFromOptions() line 188 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > > [0]PETSC ERROR: No support for this operation for this object type! > > > [0]PETSC ERROR: PC does not have apply! > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > [0]PETSC ERROR: PCApply() line 382 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > > [0]PETSC ERROR: KSPInitialResidual() line 64 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > > [0]PETSC ERROR: KSPSolve() line 446 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > > > I note that the error appears to occur at the same point. > > > > > > George > > > > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith wrote: > > > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > > > Hi Barry, > > > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > > > > > That should not happen. Please run and send all the output including the exact command line used > > > > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. > > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. > > > > > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 and send the complete error message. > > > > > > > > > Barry > > > > > > > > > > > George > > > > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > > > >> > > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > > >> > > > >>> Hi Barry, > > > >>> > > > >>> I tried with the addition of > > > >>> > > > >>> -hmpi_spawn_size 3 > > > >>> > > > >>> but I am still getting the same error though. > > > >> > > > >> The EXACT same error? Or some other error? > > > >> > > > >> What happens if you run with the -hmpi_merge_size option instead? > > > >> > > > >> Barry > > > >> > > > >> 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. > > > >> > > > >> 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > > > >> > > > >> > > > >>> I am using mpich2. Any other options to try? > > > >>> > > > >>> George > > > >>> > > > >>> > > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > > >>> > > > >>>> > > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > > >>>> > > > >>>>> Hi, > > > >>>>> > > > >>>>> I have some issues using the -pctype hmpi. I used the same setting found at > > > >>>>> > > > >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > > >>>>> > > > >>>>> i.e. > > > >>>>> -pc_type hmpi > > > >>>>> -ksp_type preonly > > > >>>>> -hmpi_ksp_type cg > > > >>>>> -hmpi_pc_type hypre > > > >>>>> -hmpi_pc_hypre_type boomeramg > > > >>>>> > > > >>>>> My command is > > > >>>>> > > > >>>>> mpiexec -n 1 myprogram > > > >>>> > > > >>>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has > > > >>>> > > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner > > > >>>> > > > >>>> This will tell you want option to start PETSc up with. > > > >>>> > > > >>>> I will fix the PC so that it prints a far more useful error message. > > > >>>> > > > >>>> > > > >>>> > > > >>>> Barry > > > >>>> > > > >>>> > > > >>>>> > > > >>>>> But, I get > > > >>>>> > > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > > >>>>> > > > >>>>> with openmpi. I get similar error with mpich2 > > > >>>>> > > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed > > > >>>>> PMPI_Bcast(1418): Invalid communicator > > > >>>>> > > > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. > > > >>>>> > > > >>>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? > > > >>>>> > > > >>>>> -- > > > >>>>> George Pau > > > >>>>> Earth Sciences Division > > > >>>>> Lawrence Berkeley National Laboratory > > > >>>>> One Cyclotron, MS 74-120 > > > >>>>> Berkeley, CA 94720 > > > >>>>> > > > >>>>> (510) 486-7196 > > > >>>>> gpau at lbl.gov > > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > -- > > > George Pau > > > Earth Sciences Division > > > Lawrence Berkeley National Laboratory > > > One Cyclotron, MS 74-120 > > > Berkeley, CA 94720 > > > > > > (510) 486-7196 > > > gpau at lbl.gov > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > -------------- next part -------------- A non-text attachment was scrubbed... Name: zstart.c Type: application/octet-stream Size: 16909 bytes Desc: not available URL: From gpau at lbl.gov Fri Aug 31 17:26:16 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 15:26:16 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: <2EABC8B5-DF3A-4B05-A49E-E26566C31979@mcs.anl.gov> References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> <2EABC8B5-DF3A-4B05-A49E-E26566C31979@mcs.anl.gov> Message-ID: hi Barry, The hmpi option is read in properly now. The error is now different when I use -hmpi_spawn_size 3 with mpiexec -n 1. My printout suggests this is now happening in the KSPSolve. George [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [0] petscinitialize_(): Running on machine: gilbert [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [0] petscinitialize_(): Running on machine: gilbert [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) [1] petscinitialize_(): Running on machine: gilbert [0] PetscHMPISpawn(): PETSc HMPI successfully spawned: number of nodes = 1 node size = 3 [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374781 max tags = 2147483647 [0] MatSetUp(): Warning not preallocating matrix storage [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781 start ksp [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781 [0] PCSetUp(): Setting up new PC [0] PetscCommDuplicate(): Duplicating a communicator -2080374782 -2080374780 max tags = 2147483647 [0] PetscCommDuplicate(): Duplicating a communicator -1006632960 -1006632959 max tags = 2147483647 [1] PetscCommDuplicate(): Duplicating a communicator -2080374779 -2080374778 max tags = 2147483647 [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 -1006632959 [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 -2080374778 [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374779 max tags = 2147483647 [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -1006632958 max tags = 2147483647 [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374779 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -1006632958 [0] VecScatterCreate(): Special case: processor zero gets entire parallel vector, rest get none [0] Petsc_DelComm(): Removing reference to PETSc communicator imbedded in a user MPI_Comm m -2080374779 [0] Petsc_DelComm(): User MPI_Comm m 1140850689 is being freed, removing reference from inner PETSc comm to this outer comm [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374779 [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374779 [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 -2080374780 [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 -1006632959 [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 -2080374778 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -1006632958 [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] MatDistribute_MPIAIJ line 192 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: [1] PCSetUp_HMPI_MP line 90 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c [1]PETSC ERROR: [1] PetscHMPIHandle line 253 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c [1]PETSC ERROR: [1] PetscHMPISpawn line 71 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 15:20:27 2012 [1]PETSC ERROR: [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib [1]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 [1]PETSC ERROR: Configure options --with-debugging=1 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 [0]PETSC ERROR: [0] MatDistribute_MPIAIJ line 192 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] PCSetUp_HMPI_MP line 90 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c [0]PETSC ERROR: [0] PetscHMPIHandle line 253 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c [0]PETSC ERROR: [0] PetscHMPISpawn line 71 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 15:20:27 2012 [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib [0]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 [0]PETSC ERROR: Configure options --with-debugging=1 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(855)........: MPI_Allreduce(sbuf=0x7fff6315ead0, rbuf=0x7fff6315eae0, count=2, MPI_INT, MPI_MAX, comm=0x84000004) failed MPIR_Allreduce_impl(712)..: MPIR_Allreduce_intra(534).: dequeue_and_set_error(596): Communication error with rank 0 On Fri, Aug 31, 2012 at 2:09 PM, Barry Smith wrote: > [see attached file: zstart.c] > > On Aug 31, 2012, at 4:07 PM, George Pau wrote: > > > Hi Barry, > > > > You forgot the file ... > > > > George > > > > On Fri, Aug 31, 2012 at 2:04 PM, Barry Smith wrote: > > > > Yikes. It is totally my fault. The handling of these merge and spawn > options is done only for PetscInitialize() for C. Not for Fortran, hence > the arguments just got ignored. > > > > Please find attached a file zstart.c put it in the directory > src/sys/ftn-custom and run make in that directory (with appropriate > PETSC_DIR and PETSC_ARCH set) > > > > Then link and run the example again. > > > > > > Barry > > > > > > On Aug 31, 2012, at 3:30 PM, George Pau wrote: > > > > > Sorry, it was a cut and paste error. I tried running the code with > all the options in the command line: > > > > > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi > -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre > boomeramg > > > > > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi > -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre > boomeramg > > > > > > but I get the exact same outputs. > > > > > > George > > > > > > > > > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith > wrote: > > > > > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > > > > > Hi Barry, > > > > > > > > For the hmpi_spawn_size, the options in my .petscrc are > > > > -info > > > > -pc_view > > > > pc_type hmpi > > > > > > How come there is no - in front of this one? > > > > > > > -ksp_type preonly > > > > -ksp_view > > > > -hmpi_pc_monitor > > > > -hmpi_ksp_monitor > > > > -hmpi_ksp_type cg > > > > -hmpi_pc_type hypre > > > > -hmpi_pc_hypre_type boomeramg > > > > -hmpi_spawn_size 3 > > > > > > > > mpiexec.mpich2 -n 1 myprogram > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: > 3978 unneeded,3222 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 360 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using > Inode routines > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > > > > > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, > root=0, comm=0x0) failed > > > > PMPI_Bcast(1418): Invalid communicator > > > > > > > > I inserted some print statement between the ksp calls and found that > the error occurs in > > > > > > > > call KSPSetFromOptions(ksp, pierr) > > > > > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my > job by > > > > > > How come there is no - in front of hmpi_merge_size 2? > > > > > > > > > Can you try putting all the arguments as command line arguments > instead of in a file? It shouldn't matter but it seems like some of the > arguments are being ignored. > > > > > > Barry > > > > > > > > > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > [1] petscinitialize_(): Running on machine: gilbert > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374780 max tags = 2147483647 > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374782 max tags = 2147483647 > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374777 max tags = 2147483647 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374780 max tags = 2147483647 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 > mallocs. > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: > 1998 unneeded,1602 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 180 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: > 1998 unneeded,1602 used > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 180 > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > > > [0] VecScatterCreate(): General case: MPI to Seq > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: > 396 unneeded,9 used > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 3 > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374782 > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: > 396 unneeded,9 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 3 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 > mallocs. > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374782 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > [0] PCSetUp(): Setting up new PC > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > > [0]PETSC ERROR: Nonconforming object sizes! > > > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named > gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > > > [0]PETSC ERROR: PCSetType() line 83 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > [0]PETSC ERROR: PCSetFromOptions() line 188 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > > [0]PETSC ERROR: No support for this operation for this object type! > > > > [0]PETSC ERROR: PC does not have apply! > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named > gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: PCApply() line 382 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > > > [0]PETSC ERROR: KSPInitialResidual() line 64 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > > > [0]PETSC ERROR: KSPSolve() line 446 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > > > > > I note that the error appears to occur at the same point. > > > > > > > > George > > > > > > > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith > wrote: > > > > > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and > MPI_Bcast. > > > > > > > > That should not happen. Please run and send all the output > including the exact command line used > > > > > > > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu > but if MPICH version that Petsc download with -download-mpich works, I can > use that. > > > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with > more than 1 cpus. But, petsc will complain that the pctype hmpi can only > be used in serial. > > > > > > > > That should not happen. Run with 2 MPI processes and > -hmpi_merge_size 2 and send the complete error message. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > > > > > >> > > > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > > > >> > > > > >>> Hi Barry, > > > > >>> > > > > >>> I tried with the addition of > > > > >>> > > > > >>> -hmpi_spawn_size 3 > > > > >>> > > > > >>> but I am still getting the same error though. > > > > >> > > > > >> The EXACT same error? Or some other error? > > > > >> > > > > >> What happens if you run with the -hmpi_merge_size > option instead? > > > > >> > > > > >> Barry > > > > >> > > > > >> 1) I am getting a crash with the spawn version I suspect is due > to bugs in the MPICH version I am using related to spawn. > > > > >> > > > > >> 2) I am getting errors with the merge version due to Apple's ASLR > which they make hard to turn off. > > > > >> > > > > >> > > > > >>> I am using mpich2. Any other options to try? > > > > >>> > > > > >>> George > > > > >>> > > > > >>> > > > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > > > >>> > > > > >>>> > > > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > > > >>>> > > > > >>>>> Hi, > > > > >>>>> > > > > >>>>> I have some issues using the -pctype hmpi. I used the same > setting found at > > > > >>>>> > > > > >>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > > > >>>>> > > > > >>>>> i.e. > > > > >>>>> -pc_type hmpi > > > > >>>>> -ksp_type preonly > > > > >>>>> -hmpi_ksp_type cg > > > > >>>>> -hmpi_pc_type hypre > > > > >>>>> -hmpi_pc_hypre_type boomeramg > > > > >>>>> > > > > >>>>> My command is > > > > >>>>> > > > > >>>>> mpiexec -n 1 myprogram > > > > >>>> > > > > >>>> Sorry the documentation doesn't make this clearer. You need to > start PETSc with special options to get the "worker" processes initialized. > From the manual page for PCHMPI it has > > > > >>>> > > > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start > up MPI for use with this preconditioner > > > > >>>> > > > > >>>> This will tell you want option to start PETSc up with. > > > > >>>> > > > > >>>> I will fix the PC so that it prints a far more useful error > message. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> Barry > > > > >>>> > > > > >>>> > > > > >>>>> > > > > >>>>> But, I get > > > > >>>>> > > > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now > abort) > > > > >>>>> > > > > >>>>> with openmpi. I get similar error with mpich2 > > > > >>>>> > > > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, > MPI_INT, root=0, comm=0x0) failed > > > > >>>>> PMPI_Bcast(1418): Invalid communicator > > > > >>>>> > > > > >>>>> I couldn't figure out what is wrong. My petsc is version > 3.3.3 and the configuration is -with-debugging=0 > --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu > machine. > > > > >>>>> > > > > >>>>> Note that with the default pc_type and ksp_type, everything is > fine. It was also tested with multiple processors. I wondering whether > there are some options that I am not specifying correctly? > > > > >>>>> > > > > >>>>> -- > > > > >>>>> George Pau > > > > >>>>> Earth Sciences Division > > > > >>>>> Lawrence Berkeley National Laboratory > > > > >>>>> One Cyclotron, MS 74-120 > > > > >>>>> Berkeley, CA 94720 > > > > >>>>> > > > > >>>>> (510) 486-7196 > > > > >>>>> gpau at lbl.gov > > > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > George Pau > > > > Earth Sciences Division > > > > Lawrence Berkeley National Laboratory > > > > One Cyclotron, MS 74-120 > > > > Berkeley, CA 94720 > > > > > > > > (510) 486-7196 > > > > gpau at lbl.gov > > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > > > > > > > -- > > > George Pau > > > Earth Sciences Division > > > Lawrence Berkeley National Laboratory > > > One Cyclotron, MS 74-120 > > > Berkeley, CA 94720 > > > > > > (510) 486-7196 > > > gpau at lbl.gov > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 31 18:38:30 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 31 Aug 2012 16:38:30 -0700 Subject: [petsc-users] Petsc crashes with intel compiler Message-ID: hi guys, After doing many many tests across different machine (Linux box and a Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in which my code crashes if I use a petsc version that is compiled with intel compiler. This does not happen if I use gcc. When I use BiCGSTAB solver along with hypre's BoomerAMG, I get the following error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: Error in HYPRE solver, error code 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCApply_HYPRE() line 160 in src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPInitialResidual() line 57 in src/ksp/ksp/interface/itres.c [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: solve() line 402 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: clear() line 438 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPInitialResidual line 45 src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Initially I thought this is somehow related to hypre, but even when I run the code with -pc_type none I get: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Infinite or not-a-number generated in norm! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: solve() line 402 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp N5PETSc9ExceptionE In both cases the solution to the linear system diverges. None of these issues happen when I build petsc with gcc. Couple of points: 1) The gcc version is clean under valgrind (except the usual PetscInitialize errors ...) but the intel versions generates all sort of errors in valgrind. Please see blow (this is during one Poisson solve) 2) This is my petsc configure option: /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 in both cases $TACC_MKL_DIR points to a mkl library compiled using intel compiler. I don't think if that has to do with any of this (looking at valgrind output) but then again that's the only version of mkl available on the cluster (Lonestar). Do you think this is somehow a compiler bug(?) or something more like an undefined behavior in my own code? ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x8283154: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x8283136: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:332) ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) ==8404== Solving a nonlinear PB to get steady state solution ... ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR (IJMatrix_parcsr.c:179) ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes (HYPRE_IJMatrix.c:818) ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:48) ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, hypre_IJMatrix_struct**) (mhyp.c:80) ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== ==8404== Syscall param mmap(length) contains uninitialised byte(s) ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize (aux_parcsr_matrix.c:177) ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:273) ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1685) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x828313B: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x8283145: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1354) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:372) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1767) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Use of uninitialised value of size 8 ==8404== at 0x8283131: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1355) ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) ==8404== ==8404== Conditional jump or move depends on uninitialised value(s) ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) (vecstash.c:37) ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, double const*) (pbvec.c:207) ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, KSPConvergedReason*, void*) (iterativ.c:585) ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:446) ==8404== -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 31 18:51:02 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 18:51:02 -0500 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: Can you remove pthreadclasses from your configure? If the problem is still there, it looks a bit like a misconfigured MPI. But get rid of the pthreadclasses first. On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh wrote: > hi guys, > > After doing many many tests across different machine (Linux box and a Mac) > and clusters (Gordon and Lonestar) I'm able to reproduce a test in which my > code crashes if I use a petsc version that is compiled with intel compiler. > This does not happen if I use gcc. When I use BiCGSTAB solver along with > hypre's BoomerAMG, I get the following error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE solver, error code 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApply_HYPRE() line 160 in src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPInitialResidual() line 57 in > src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: clear() line 438 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c > [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown > [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPInitialResidual line 45 > src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > Initially I thought this is somehow related to hypre, but even when I run > the code with -pc_type none I get: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > N5PETSc9ExceptionE > > > In both cases the solution to the linear system diverges. None of these > issues happen when I build petsc with gcc. > > Couple of points: > > 1) The gcc version is clean under valgrind (except the usual > PetscInitialize errors ...) but the intel versions generates all sort of > errors in valgrind. Please see blow (this is during one Poisson solve) > 2) This is my petsc configure option: > > /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx > --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME > --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx > --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME > --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > > in both cases $TACC_MKL_DIR points to a mkl library compiled using intel > compiler. I don't think if that has to do with any of this (looking at > valgrind output) but then again that's the only version of mkl available on > the cluster (Lonestar). > > Do you think this is somehow a compiler bug(?) or something more like an > undefined behavior in my own code? > > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283154: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283136: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) > ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) > ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) > ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) > ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) > ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, char > const*, char const*) (petscLinearSolver.cpp:332) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() > (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > Solving a nonlinear PB to get steady state solution ... > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR > (IJMatrix_parcsr.c:179) > ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes > (HYPRE_IJMatrix.c:818) > ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:48) > ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, > hypre_IJMatrix_struct**) (mhyp.c:80) > ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Syscall param mmap(length) contains uninitialised byte(s) > ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) > ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) > (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) > (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char > const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char > const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char > const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char > const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char > const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char > const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize > (aux_parcsr_matrix.c:177) > ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:273) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1685) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, > int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) > ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) > ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x828313B: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283145: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1354) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) > ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:372) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) > ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1767) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283131: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator > (par_rap.c:1355) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, > PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, > char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char > const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) > (vecstash.c:37) > ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, > double const*) (pbvec.c:207) > ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) > ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) > ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, > KSPConvergedReason*, void*) (iterativ.c:585) > ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) > ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:446) > ==8404== > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 31 18:53:55 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 31 Aug 2012 16:53:55 -0700 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: I'll try it right away On Fri, Aug 31, 2012 at 4:51 PM, Jed Brown wrote: > Can you remove pthreadclasses from your configure? If the problem is still > there, it looks a bit like a misconfigured MPI. But get rid of the > pthreadclasses first. > > > On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh wrote: > >> hi guys, >> >> After doing many many tests across different machine (Linux box and a >> Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in >> which my code crashes if I use a petsc version that is compiled with intel >> compiler. This does not happen if I use gcc. When I use BiCGSTAB solver >> along with hypre's BoomerAMG, I get the following error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Error in HYPRE solver, error code 1! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCApply_HYPRE() line 160 in src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPInitialResidual() line 57 in >> src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve() line 402 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in >> src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: clear() line 438 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c >> [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >> [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown >> [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 >> src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >> [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPInitialResidual line 45 >> src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> >> >> Initially I thought this is somehow related to hypre, but even when I run >> the code with -pc_type none I get: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Floating point exception! >> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve() line 402 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> N5PETSc9ExceptionE >> >> >> In both cases the solution to the linear system diverges. None of these >> issues happen when I build petsc with gcc. >> >> Couple of points: >> >> 1) The gcc version is clean under valgrind (except the usual >> PetscInitialize errors ...) but the intel versions generates all sort of >> errors in valgrind. Please see blow (this is during one Poisson solve) >> 2) This is my petsc configure option: >> >> /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx >> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >> >> ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx >> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >> >> >> in both cases $TACC_MKL_DIR points to a mkl library compiled using intel >> compiler. I don't think if that has to do with any of this (looking at >> valgrind output) but then again that's the only version of mkl available on >> the cluster (Lonestar). >> >> Do you think this is somehow a compiler bug(?) or something more like an >> undefined behavior in my own code? >> >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283154: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283136: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) >> ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) >> ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) >> ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) >> ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) >> ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:332) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> Solving a nonlinear PB to get steady state solution ... >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR >> (IJMatrix_parcsr.c:179) >> ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes >> (HYPRE_IJMatrix.c:818) >> ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:48) >> ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, >> hypre_IJMatrix_struct**) (mhyp.c:80) >> ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Syscall param mmap(length) contains uninitialised byte(s) >> ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) >> ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize >> (aux_parcsr_matrix.c:177) >> ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:273) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >> ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1685) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) >> ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) >> ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x828313B: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283145: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1354) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) >> ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:372) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >> ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) >> ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1767) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283131: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1355) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) >> (vecstash.c:37) >> ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, >> double const*) (pbvec.c:207) >> ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) >> ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) >> ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, >> KSPConvergedReason*, void*) (iterativ.c:585) >> ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) >> ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:446) >> ==8404== >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 31 19:12:27 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 31 Aug 2012 19:12:27 -0500 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 6:51 PM, Jed Brown wrote: > Can you remove pthreadclasses from your configure? If the problem is still > there, it looks a bit like a misconfigured MPI. But get rid of the > pthreadclasses first. After that, I would get rid of MKL in favor of --download-f-blas-lapack. My guess is that MKL is doing something crazy in the compiler. Matt > On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh wrote: > >> hi guys, >> >> After doing many many tests across different machine (Linux box and a >> Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in >> which my code crashes if I use a petsc version that is compiled with intel >> compiler. This does not happen if I use gcc. When I use BiCGSTAB solver >> along with hypre's BoomerAMG, I get the following error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Error in HYPRE solver, error code 1! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCApply_HYPRE() line 160 in src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPInitialResidual() line 57 in >> src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve() line 402 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in >> src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: clear() line 438 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c >> [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >> [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown >> [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 >> src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >> [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPInitialResidual line 45 >> src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> >> >> Initially I thought this is somehow related to hypre, but even when I run >> the code with -pc_type none I get: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Floating point exception! >> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 >> CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 >> [0]PETSC ERROR: Libraries linked from >> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >> --with-clanguage=cxx --with-pthreadclasses=1 >> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >> --download-superlu_dist=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c >> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve() line 402 in >> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >> N5PETSc9ExceptionE >> >> >> In both cases the solution to the linear system diverges. None of these >> issues happen when I build petsc with gcc. >> >> Couple of points: >> >> 1) The gcc version is clean under valgrind (except the usual >> PetscInitialize errors ...) but the intel versions generates all sort of >> errors in valgrind. Please see blow (this is during one Poisson solve) >> 2) This is my petsc configure option: >> >> /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx >> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >> >> ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx >> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >> >> >> in both cases $TACC_MKL_DIR points to a mkl library compiled using intel >> compiler. I don't think if that has to do with any of this (looking at >> valgrind output) but then again that's the only version of mkl available on >> the cluster (Lonestar). >> >> Do you think this is somehow a compiler bug(?) or something more like an >> undefined behavior in my own code? >> >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283154: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283136: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:331) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) >> ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) >> ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) >> ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) >> ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) >> ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, >> char const*, char const*) (petscLinearSolver.cpp:332) >> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >> (pnpSolver_2d_FVM.cpp:46) >> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >> ==8404== >> Solving a nonlinear PB to get steady state solution ... >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR >> (IJMatrix_parcsr.c:179) >> ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes >> (HYPRE_IJMatrix.c:818) >> ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:48) >> ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, >> hypre_IJMatrix_struct**) (mhyp.c:80) >> ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Syscall param mmap(length) contains uninitialised byte(s) >> ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) >> ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) >> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >> (bvec2.c:887) >> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >> (matrix.c:8149) >> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >> (IJVector_parcsr.c:124) >> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >> hypre_IJVector_struct**) (vhyp.c:21) >> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >> const*, char const*) (mal.c:75) >> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char >> const*, char const*) (mtr.c:322) >> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >> (par_csr_matrix.c:200) >> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:272) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize >> (aux_parcsr_matrix.c:177) >> ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR >> (IJMatrix_parcsr.c:273) >> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:211) >> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >> hypre_IJMatrix_struct*) (mhyp.c:122) >> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >> ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1012) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1122) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >> ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1685) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >> int) (pbSolver_2d_FVM.cpp:148) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) >> ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) >> ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x828313B: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) >> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >> (par_coarsen.c:1918) >> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283145: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1354) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) >> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) >> ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:372) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >> ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) >> ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1767) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Use of uninitialised value of size 8 >> ==8404== at 0x8283131: __intel_new_memset (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x8261535: _intel_fast_memset.J (in >> /opt/apps/intel/11.1/lib/intel64/libirc.so) >> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >> ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator >> (par_rap.c:1355) >> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:402) >> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >> PetscBool) (petscLinearSolver.cpp:402) >> ==8404== >> ==8404== Conditional jump or move depends on uninitialised value(s) >> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >> const*, char const*, char const*, void**) (mal.c:30) >> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >> const*, char const*, char const*, void**) (mtr.c:190) >> ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) >> (vecstash.c:37) >> ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, >> double const*) (pbvec.c:207) >> ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) >> ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) >> ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, >> KSPConvergedReason*, void*) (iterativ.c:585) >> ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) >> ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >> (itfunc.c:446) >> ==8404== >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Fri Aug 31 19:27:06 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Fri, 31 Aug 2012 20:27:06 -0400 Subject: [petsc-users] MG Preconditioning Message-ID: Hi I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to use PCMG to precondition a linear system of equations resulting from the discretization of the EUler equations. It is from a multi-block code so I have the ability to easily generate restriction operators using geometric method. I'm currently trying to setup a simple 2V cycle. For this I use: ! Create the restriction operator between finest level and one below call createRestrictOperator(RL1, 1) call PCSetFromOptions(pc, PETScIerr) nlevels=2 !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't work from fortran!!! call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr) call PCMGSetRestriction(pc, 1, RL1, PETScierr) call PCView(pc, PETScierr) I run with the command line argument -pc_mg_levels 2 . First issue: PCMGSetLevels does not work correctly from Fortran. The traceback I get when I run (with the commented-out line above and without the command line arguments) is: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 144 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c [0]PETSC ERROR: [0] KSPCreate line 557 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c [0]PETSC ERROR: [0] PCMGSetLevels line 180 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c I can get around that issue by using -pc_mg_levels 2 and not calling the PCMGSetLevels() command, but I would like to actually use this command and not have set it via manually setting an option. The result of the PCView is listed below which appears to be what I expect (given the default options) PC Object: 1 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=2 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 1 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0, max = 0 maximum iterations=2, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix followed by preconditioner matrix: Matrix Object: 1 MPI processes type: seqbaij rows=92160, cols=92160, bs=5 total: nonzeros=5582400, allocated nonzeros=5582400 total number of mallocs used during MatSetValues calls =0 block size is 5 Matrix Object: 1 MPI processes type: seqbaij rows=92160, cols=92160, bs=5 total: nonzeros=3089600, allocated nonzeros=3089600 total number of mallocs used during MatSetValues calls =0 block size is 5 So that's ok. When it gets to actually solving my system with KSPSolve() I get a returned error code 73 which is "object in argument is in wrong state, e.g. unassembled mat" and the traceback is: [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatGetVecs line 8142 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: [0] KSPGetVecs line 774 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: [0] PCSetUp_MG line 508 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCSetUp line 810 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 182 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPSolve line 351 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c Which would indicate to me its an issue with one of the matrices involved, either the linear operator, A, the preconditioner P or my provided restriction matrix RL1. A and P i'm pretty sure are OK, since it works just fine when I use my usual BlockJacobi/Additive Schwartz with ILU(k) on the subdomains. Which leaves the Restriction operator as the issue. I added the following code before PCView() call MatGetVecs(RL1, vec1, vec2, ierr) call VecSet(vec1, 1.0, ierr) call MatMult(RL1, vec1, vec2, ierr) which ran through just fine (I check each of the ierrs but those are not shown). So I'm at a loss as to what is causing the error. I'm using an up-to-date copy of petsc-3.3 from the repository. Thanks, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 31 19:46:16 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 31 Aug 2012 17:46:16 -0700 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: Alright, just removed pthreadclasses and it did not help. The code still crashes. Matt, I'm gonna try MKL now; However, don't you think its strange that the gcc version works even when I use MKL? On Fri, Aug 31, 2012 at 5:12 PM, Matthew Knepley wrote: > On Fri, Aug 31, 2012 at 6:51 PM, Jed Brown wrote: > >> Can you remove pthreadclasses from your configure? If the problem is >> still there, it looks a bit like a misconfigured MPI. But get rid of the >> pthreadclasses first. > > > After that, I would get rid of MKL in favor of --download-f-blas-lapack. > My guess is that MKL is doing something > crazy in the compiler. > > Matt > > >> On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh wrote: >> >>> hi guys, >>> >>> After doing many many tests across different machine (Linux box and a >>> Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in >>> which my code crashes if I use a petsc version that is compiled with intel >>> compiler. This does not happen if I use gcc. When I use BiCGSTAB solver >>> along with hypre's BoomerAMG, I get the following error message: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Error in external library! >>> [0]PETSC ERROR: Error in HYPRE solver, error code 1! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>> 15:42:00 CDT 2012 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>> --with-clanguage=cxx --with-pthreadclasses=1 >>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>> --download-superlu_dist=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PCApply_HYPRE() line 160 in >>> src/ksp/pc/impls/hypre/hypre.c >>> [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPInitialResidual() line 57 in >>> src/ksp/ksp/interface/itres.c >>> [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: solve() line 402 in >>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Error in external library! >>> [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>> 15:42:00 CDT 2012 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>> --with-clanguage=cxx --with-pthreadclasses=1 >>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>> --download-superlu_dist=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in >>> src/ksp/pc/impls/hypre/hypre.c >>> [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: clear() line 438 in >>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >>> corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c >>> [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c >>> [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>> [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown >>> [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 >>> src/ksp/pc/impls/hypre/hypre.c >>> [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>> [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c >>> [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: [0] KSPInitialResidual line 45 >>> src/ksp/ksp/interface/itres.c >>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c >>> [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>> 15:42:00 CDT 2012 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>> --with-clanguage=cxx --with-pthreadclasses=1 >>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>> --download-superlu_dist=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>> unknown file >>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>> >>> >>> Initially I thought this is somehow related to hypre, but even when I >>> run the code with -pc_type none I get: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Floating point exception! >>> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>> 15:42:00 CDT 2012 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>> --with-clanguage=cxx --with-pthreadclasses=1 >>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>> --download-superlu_dist=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: solve() line 402 in >>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>> N5PETSc9ExceptionE >>> >>> >>> In both cases the solution to the linear system diverges. None of these >>> issues happen when I build petsc with gcc. >>> >>> Couple of points: >>> >>> 1) The gcc version is clean under valgrind (except the usual >>> PetscInitialize errors ...) but the intel versions generates all sort of >>> errors in valgrind. Please see blow (this is during one Poisson solve) >>> 2) This is my petsc configure option: >>> >>> /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx >>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>> >>> ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx >>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>> >>> >>> in both cases $TACC_MKL_DIR points to a mkl library compiled using intel >>> compiler. I don't think if that has to do with any of this (looking at >>> valgrind output) but then again that's the only version of mkl available on >>> the cluster (Lonestar). >>> >>> Do you think this is somehow a compiler bug(?) or something more like an >>> undefined behavior in my own code? >>> >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x8283154: __intel_new_memset (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x8283136: __intel_new_memset (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:331) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) >>> ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) >>> ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) >>> ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) >>> ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) >>> ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, >>> char const*, char const*) (petscLinearSolver.cpp:332) >>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>> (pnpSolver_2d_FVM.cpp:46) >>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>> ==8404== >>> Solving a nonlinear PB to get steady state solution ... >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR >>> (IJMatrix_parcsr.c:179) >>> ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes >>> (HYPRE_IJMatrix.c:818) >>> ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:48) >>> ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, >>> hypre_IJMatrix_struct**) (mhyp.c:80) >>> ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== >>> ==8404== Syscall param mmap(length) contains uninitialised byte(s) >>> ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) >>> ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) >>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>> (bvec2.c:887) >>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>> (matrix.c:8149) >>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>> (IJVector_parcsr.c:124) >>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>> hypre_IJVector_struct**) (vhyp.c:21) >>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>> (IJVector_parcsr.c:124) >>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>> hypre_IJVector_struct**) (vhyp.c:21) >>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>> (IJVector_parcsr.c:124) >>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) >>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>> hypre_IJVector_struct**) (vhyp.c:21) >>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>> const*, char const*) (mal.c:75) >>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>> char const*, char const*) (mtr.c:322) >>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) >>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>> const*, char const*) (mal.c:75) >>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>> char const*, char const*) (mtr.c:322) >>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>> const*, char const*) (mal.c:75) >>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>> char const*, char const*) (mtr.c:322) >>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>> (par_csr_matrix.c:200) >>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:272) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize >>> (aux_parcsr_matrix.c:177) >>> ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR >>> (IJMatrix_parcsr.c:273) >>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) >>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:211) >>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>> hypre_IJMatrix_struct*) (mhyp.c:122) >>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>> ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>> (par_coarsen.c:1918) >>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>> (par_coarsen.c:1918) >>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1012) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1012) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1012) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1122) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1122) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1122) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) >>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>> ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1685) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>> int) (pbSolver_2d_FVM.cpp:148) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) >>> ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) >>> ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x828313B: __intel_new_memset (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) >>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>> (par_coarsen.c:1918) >>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x8283145: __intel_new_memset (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1354) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) >>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) >>> ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:372) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>> ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) >>> ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1767) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Use of uninitialised value of size 8 >>> ==8404== at 0x8283131: __intel_new_memset (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>> ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator >>> (par_rap.c:1355) >>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:402) >>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>> PetscBool) (petscLinearSolver.cpp:402) >>> ==8404== >>> ==8404== Conditional jump or move depends on uninitialised value(s) >>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>> const*, char const*, char const*, void**) (mal.c:30) >>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>> const*, char const*, char const*, void**) (mtr.c:190) >>> ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) >>> (vecstash.c:37) >>> ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, >>> double const*) (pbvec.c:207) >>> ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) >>> ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) >>> ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, >>> KSPConvergedReason*, void*) (iterativ.c:585) >>> ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) >>> ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>> (itfunc.c:446) >>> ==8404== >>> >>> >>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Fri Aug 31 20:16:41 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Fri, 31 Aug 2012 20:16:41 -0500 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: <504161F9.2070605@geology.wisc.edu> You can try the configure flags used by PETSc (3.2) that is pre-installed on Lonestar (configured with Hypre, MKL, Intel MPI etc.). $ module load petsc and then try any of the PETSc examples with bcgs hypre/none On 08/31/2012 07:46 PM, Mohammad Mirzadeh wrote: > Alright, just removed pthreadclasses and it did not help. The code > still crashes. > > Matt, I'm gonna try MKL now; However, don't you think its strange that > the gcc version works even when I use MKL? > > On Fri, Aug 31, 2012 at 5:12 PM, Matthew Knepley > wrote: > > On Fri, Aug 31, 2012 at 6:51 PM, Jed Brown > wrote: > > Can you remove pthreadclasses from your configure? If the > problem is still there, it looks a bit like a misconfigured > MPI. But get rid of the pthreadclasses first. > > > After that, I would get rid of MKL in favor of > --download-f-blas-lapack. My guess is that MKL is doing something > crazy in the compiler. > > Matt > > On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh > > wrote: > > hi guys, > > After doing many many tests across different machine > (Linux box and a Mac) and clusters (Gordon and Lonestar) > I'm able to reproduce a test in which my code crashes if I > use a petsc version that is compiled with intel compiler. > This does not happen if I use gcc. When I use BiCGSTAB > solver along with hypre's BoomerAMG, I get the following > error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE solver, error code 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri > Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent > updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble > shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu > by mmirzade Fri Aug > 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options > PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx > --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl > --download-hypre=1 --download-hdf5=1 --download-metis=1 > --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApply_HYPRE() line 160 in > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCApply() line 384 in > src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPInitialResidual() line 57 in > src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: KSPSolve_BCGS() line 65 in > src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri > Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent > updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble > shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu > by mmirzade Fri Aug > 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options > PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx > --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl > --download-hypre=1 --download-hdf5=1 --download-metis=1 > --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCDestroy() line 119 in > src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPDestroy() line 786 in > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: clear() line 438 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack > below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack > are not available, > [0]PETSC ERROR: INSTEAD the line number of the start > of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecDestroy line 574 > src/vec/vec/interface/vector.c > [0]PETSC ERROR: [0] KSPReset_BCGS line 194 > src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPReset line 729 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPDestroy line 768 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 > unknownunknown > [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCDestroy line 110 > src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPDestroy line 768 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] PCApply_HYPRE line 149 > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCApply line 373 > src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPInitialResidual line 45 > src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 > src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPSolve line 351 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri > Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent > updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble > shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu > by mmirzade Fri Aug > 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options > PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx > --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl > --download-hypre=1 --download-hdf5=1 --download-metis=1 > --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown > directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > Initially I thought this is somehow related to hypre, but > even when I run the code with -pc_type none I get: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri > Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent > updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble > shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named > c341-116.ls4.tacc.utexas.edu > by mmirzade Fri Aug > 31 18:15:04 2012 > [0]PETSC ERROR: Libraries linked from > /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options > PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx > --with-pthreadclasses=1 > --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 > --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl > --download-hypre=1 --download-hdf5=1 --download-metis=1 > --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 169 in > src/vec/vec/interface/rvector.c > [0]PETSC ERROR: KSPSolve_BCGS() line 78 in > src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in > "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > N5PETSc9ExceptionE > > > In both cases the solution to the linear system diverges. > None of these issues happen when I build petsc with gcc. > > Couple of points: > > 1) The gcc version is clean under valgrind (except the > usual PetscInitialize errors ...) but the intel versions > generates all sort of errors in valgrind. Please see blow > (this is during one Poisson solve) > 2) This is my petsc configure option: > > /configure PETSC_ARCH=arch-linux-gcc-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=$MPICH_HOME > --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > > ./configure PETSC_ARCH=arch-linux-intel-debug > --with-clanguage=cxx --with-pthreadclasses=1 > --with-mpi-dir=$MPICH_HOME > --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 > --download-hdf5=1 --download-metis=1 --download-parmetis=1 > --download-superlu_dist=1 > > > in both cases $TACC_MKL_DIR points to a mkl library > compiled using intel compiler. I don't think if that has > to do with any of this (looking at valgrind output) but > then again that's the only version of mkl available on the > cluster (Lonestar). > > Do you think this is somehow a compiler bug(?) or > something more like an undefined behavior in my own code? > > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283154: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283136: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe > (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc > (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) > (pcset.c:83) > ==8404== by 0x46B3FE: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14E6385: hypre_BoomerAMGCreate > (par_amg.c:203) > ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate > (HYPRE_parcsr_amg.c:31) > ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) > ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) > (hypre.c:895) > ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) > (pcset.c:196) > ==8404== by 0x46B487: > PetscLinearSolver::setupSolver(MatStructure, char const*, > char const*) (petscLinearSolver.cpp:332) > ==8404== by 0x46E7D1: > amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > Solving a nonlinear PB to get steady state solution ... > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14DB097: > hypre_IJMatrixSetDiagOffdSizesParCSR (IJMatrix_parcsr.c:179) > ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes > (HYPRE_IJMatrix.c:818) > ==8404== by 0x11A2728: > MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:48) > ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, > hypre_IJMatrix_struct**) (mhyp.c:80) > ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Syscall param mmap(length) contains uninitialised > byte(s) > ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so > ) > ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, > double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) > (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, > _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize > (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize > (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize > (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize > (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize > (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize > (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize > (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize > (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar > (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize > (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, > hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char > const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, > char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char > const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, > char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char > const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, > char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize > (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize > (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize > (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize > (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize > (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize > (aux_parcsr_matrix.c:177) > ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR > (IJMatrix_parcsr.c:273) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize > (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: > MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, > hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize > (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize > (par_vector.c:150) > ==8404== by 0x14ECF47: hypre_BoomerAMGSetup > (par_amg_setup.c:464) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge > (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup > (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge > (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup > (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x153F2EA: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1685) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: > amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) > (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate > (par_csr_matrix.c:60) > ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS > (par_strength.c:156) > ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup > (par_amg_setup.c:736) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x828313B: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge > (par_coarsen.c:988) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout > (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup > (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283145: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2B7: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1354) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3457: hypre_CSRMatrixTranspose > (csr_matop.c:354) > ==8404== by 0x153B7B8: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:372) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy > (csr_matrix.c:69) > ==8404== by 0x153F6C2: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1767) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283131: __intel_new_memset (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in > /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2DD: > hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1355) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup > (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup > (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: > PetscLinearSolver::solve(ArrayV&, PetscBool) > (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised > value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, > int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned > long, int, char const*, char const*, char const*, void**) > (mtr.c:190) > ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, > VecStash*) (vecstash.c:37) > ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, > PetscBool, int, double const*) (pbvec.c:207) > ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, > _p_Vec**) (pbvec.c:70) > ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) > (vector.c:551) > ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, > double, KSPConvergedReason*, void*) (iterativ.c:585) > ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) > ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, > _p_Vec*) (itfunc.c:446) > ==8404== > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to > which their experiments lead. > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Aug 31 20:45:29 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 31 Aug 2012 18:45:29 -0700 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: <504161F9.2070605@geology.wisc.edu> References: <504161F9.2070605@geology.wisc.edu> Message-ID: Alright. it was not MKL either! Tabrez, I guess I could do that ... I just want to be sure this is a nasty bug in compiler/mpi and/or undefined behavior in my code; those things are quite nasty. Should I try a different MPI implementation? Right now its mvapich2/1.6 On Fri, Aug 31, 2012 at 6:16 PM, Tabrez Ali wrote: > You can try the configure flags used by PETSc (3.2) that is pre-installed > on Lonestar (configured with Hypre, MKL, Intel MPI etc.). > > $ module load petsc > > and then try any of the PETSc examples with bcgs hypre/none > > On 08/31/2012 07:46 PM, Mohammad Mirzadeh wrote: > > Alright, just removed pthreadclasses and it did not help. The code still > crashes. > > Matt, I'm gonna try MKL now; However, don't you think its strange that > the gcc version works even when I use MKL? > > On Fri, Aug 31, 2012 at 5:12 PM, Matthew Knepley wrote: > >> On Fri, Aug 31, 2012 at 6:51 PM, Jed Brown wrote: >> >>> Can you remove pthreadclasses from your configure? If the problem is >>> still there, it looks a bit like a misconfigured MPI. But get rid of the >>> pthreadclasses first. >> >> >> After that, I would get rid of MKL in favor of >> --download-f-blas-lapack. My guess is that MKL is doing something >> crazy in the compiler. >> >> Matt >> >> >>> On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh >> > wrote: >>> >>>> hi guys, >>>> >>>> After doing many many tests across different machine (Linux box and a >>>> Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in >>>> which my code crashes if I use a petsc version that is compiled with intel >>>> compiler. This does not happen if I use gcc. When I use BiCGSTAB solver >>>> along with hypre's BoomerAMG, I get the following error message: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Error in external library! >>>> [0]PETSC ERROR: Error in HYPRE solver, error code 1! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>> 15:42:00 CDT 2012 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>> --download-superlu_dist=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: PCApply_HYPRE() line 160 in >>>> src/ksp/pc/impls/hypre/hypre.c >>>> [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: KSPInitialResidual() line 57 in >>>> src/ksp/ksp/interface/itres.c >>>> [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c >>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: solve() line 402 in >>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Error in external library! >>>> [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>> 15:42:00 CDT 2012 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>> --download-superlu_dist=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in >>>> src/ksp/pc/impls/hypre/hypre.c >>>> [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: clear() line 438 in >>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>> probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> [0]PETSC ERROR: or see >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >>>> corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c >>>> [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>>> [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown >>>> [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 >>>> src/ksp/pc/impls/hypre/hypre.c >>>> [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>>> [0]PETSC ERROR: [0] PCApply_HYPRE line 149 >>>> src/ksp/pc/impls/hypre/hypre.c >>>> [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: [0] KSPInitialResidual line 45 >>>> src/ksp/ksp/interface/itres.c >>>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c >>>> [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Signal received! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>> 15:42:00 CDT 2012 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>> --download-superlu_dist=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>>> unknown file >>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>> >>>> >>>> Initially I thought this is somehow related to hypre, but even when I >>>> run the code with -pc_type none I get: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Floating point exception! >>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>> 15:42:00 CDT 2012 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>> --download-superlu_dist=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c >>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: solve() line 402 in >>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>> N5PETSc9ExceptionE >>>> >>>> >>>> In both cases the solution to the linear system diverges. None of >>>> these issues happen when I build petsc with gcc. >>>> >>>> Couple of points: >>>> >>>> 1) The gcc version is clean under valgrind (except the usual >>>> PetscInitialize errors ...) but the intel versions generates all sort of >>>> errors in valgrind. Please see blow (this is during one Poisson solve) >>>> 2) This is my petsc configure option: >>>> >>>> /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx >>>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>>> >>>> ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx >>>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>>> >>>> >>>> in both cases $TACC_MKL_DIR points to a mkl library compiled using >>>> intel compiler. I don't think if that has to do with any of this (looking >>>> at valgrind output) but then again that's the only version of mkl available >>>> on the cluster (Lonestar). >>>> >>>> Do you think this is somehow a compiler bug(?) or something more like >>>> an undefined behavior in my own code? >>>> >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x8283154: __intel_new_memset (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x8283136: __intel_new_memset (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) >>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) >>>> ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) >>>> ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) >>>> ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) >>>> ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) >>>> ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, >>>> char const*, char const*) (petscLinearSolver.cpp:332) >>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>> (pnpSolver_2d_FVM.cpp:46) >>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>> ==8404== >>>> Solving a nonlinear PB to get steady state solution ... >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR >>>> (IJMatrix_parcsr.c:179) >>>> ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes >>>> (HYPRE_IJMatrix.c:818) >>>> ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, >>>> _p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:48) >>>> ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, >>>> hypre_IJMatrix_struct**) (mhyp.c:80) >>>> ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== >>>> ==8404== Syscall param mmap(length) contains uninitialised byte(s) >>>> ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) >>>> ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) >>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>> (bvec2.c:887) >>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>> (matrix.c:8149) >>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>> (IJVector_parcsr.c:124) >>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>> (HYPRE_IJVector.c:229) >>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>> hypre_IJVector_struct**) (vhyp.c:21) >>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>> (IJVector_parcsr.c:124) >>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>> (HYPRE_IJVector.c:229) >>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>> hypre_IJVector_struct**) (vhyp.c:21) >>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>> (IJVector_parcsr.c:124) >>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>> (HYPRE_IJVector.c:229) >>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>> hypre_IJVector_struct**) (vhyp.c:21) >>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>> const*, char const*) (mal.c:75) >>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>> char const*, char const*) (mtr.c:322) >>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) >>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>> const*, char const*) (mal.c:75) >>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>> char const*, char const*) (mtr.c:322) >>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>> const*, char const*) (mal.c:75) >>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>> char const*, char const*) (mtr.c:322) >>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>> (par_csr_matrix.c:200) >>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:272) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize >>>> (aux_parcsr_matrix.c:177) >>>> ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR >>>> (IJMatrix_parcsr.c:273) >>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>> (HYPRE_IJMatrix.c:303) >>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>> ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>> (par_coarsen.c:1918) >>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) >>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>> (par_coarsen.c:1918) >>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1012) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1012) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1012) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1122) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1122) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1122) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) >>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>>> ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1685) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, >>>> int) (pbSolver_2d_FVM.cpp:148) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) >>>> ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) >>>> ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x828313B: __intel_new_memset (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) >>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>> (par_coarsen.c:1918) >>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x8283145: __intel_new_memset (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1354) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) >>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) >>>> ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:372) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>>> ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) >>>> ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1767) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Use of uninitialised value of size 8 >>>> ==8404== at 0x8283131: __intel_new_memset (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>> ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator >>>> (par_rap.c:1355) >>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:402) >>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>> PetscBool) (petscLinearSolver.cpp:402) >>>> ==8404== >>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mal.c:30) >>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>> const*, char const*, char const*, void**) (mtr.c:190) >>>> ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) >>>> (vecstash.c:37) >>>> ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, >>>> double const*) (pbvec.c:207) >>>> ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) >>>> (pbvec.c:70) >>>> ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) >>>> ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, >>>> KSPConvergedReason*, void*) (iterativ.c:585) >>>> ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) >>>> ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>> (itfunc.c:446) >>>> ==8404== >>>> >>>> >>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 31 20:48:18 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 20:48:18 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: src/ksp/pc/examples/tests/ex8f.F is a simple example, but I made the modification below and it runs fine. diff --git a/src/ksp/pc/examples/tests/ex8f.F b/src/ksp/pc/examples/tests/ex8f.F --- a/src/ksp/pc/examples/tests/ex8f.F +++ b/src/ksp/pc/examples/tests/ex8f.F @@ -35,7 +35,7 @@ Vec x,b,u PC pc PetscInt n,dim,istart,iend - PetscInt i,j,jj,ii,one,zero + PetscInt i,j,jj,ii,one,two,zero PetscErrorCode ierr PetscScalar v external MyResidual @@ -50,6 +50,7 @@ pfive = .5d0 n = 6 dim = n*n + two = 2 one = 1 zero = 0 @@ -138,7 +139,7 @@ call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc,PCMG,ierr) - call PCMGSetLevels(pc,one,PETSC_NULL_OBJECT,ierr) + call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) call PCMGSetResidual(pc,zero,PCMGDefaultResidual,A,ierr) call PCMGSetResidual(pc,zero,MyResidual,A,ierr) @@ -147,6 +148,7 @@ ! also serves as the preconditioning matrix. call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr) + call KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD,ierr) call KSPDestroy(ksp,ierr) $ mpirun.hydra -n 2 ./ex8f KSP Object: 2 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 2 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=2 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 2 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 2 MPI processes type: redundant Redundant preconditioner: Not yet setup Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 2 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0, max = 0 maximum iterations=2, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_1_) 2 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 2 MPI processes type: mpiaij rows=36, cols=36 total: nonzeros=156, allocated nonzeros=252 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines On Fri, Aug 31, 2012 at 7:27 PM, Gaetan Kenway wrote: > Hi > > I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to > use PCMG to precondition a linear system of equations resulting from the > discretization of the EUler equations. It is from a multi-block code so I > have the ability to easily generate restriction operators using geometric > method. > > I'm currently trying to setup a simple 2V cycle. For this I use: > > ! Create the restriction operator between finest level and one below > call createRestrictOperator(RL1, 1) > > call PCSetFromOptions(pc, PETScIerr) > nlevels=2 > !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't > work from fortran!!! > call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr) > call PCMGSetRestriction(pc, 1, RL1, PETScierr) > > call PCView(pc, PETScierr) > > I run with the command line argument -pc_mg_levels 2 . > > First issue: PCMGSetLevels does not work correctly from Fortran. The > traceback I get when I run (with the commented-out line above and without > the command line arguments) is: > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscCommDuplicate line 144 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c > [0]PETSC ERROR: [0] KSPCreate line 557 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c > [0]PETSC ERROR: [0] PCMGSetLevels line 180 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c > > I can get around that issue by using -pc_mg_levels 2 and not calling the > PCMGSetLevels() command, but I would like to actually use this command and > not have set it via manually setting an option. The result of the PCView is > listed below which appears to be what I expect (given the default options) > > PC Object: 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 1 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 1 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0, max = 0 > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix followed by preconditioner matrix: > Matrix Object: 1 MPI processes > type: seqbaij > rows=92160, cols=92160, bs=5 > total: nonzeros=5582400, allocated nonzeros=5582400 > total number of mallocs used during MatSetValues calls =0 > block size is 5 > Matrix Object: 1 MPI processes > type: seqbaij > rows=92160, cols=92160, bs=5 > total: nonzeros=3089600, allocated nonzeros=3089600 > total number of mallocs used during MatSetValues calls =0 > block size is 5 > > > So that's ok. When it gets to actually solving my system with KSPSolve() I > get a returned error code 73 which is "object in argument is in wrong > state, e.g. unassembled mat" > In general, please don't paraphrase error messages. With these problems, try running in a debugger to get a real trace and with valgrind. The most common problem is stack corruption due to passing the wrong number or types of arguments. > > and the traceback is: > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatGetVecs line 8142 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] KSPGetVecs line 774 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: [0] PCSetUp_MG line 508 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCSetUp line 810 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 182 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 351 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > > Which would indicate to me its an issue with one of the matrices involved, > either the linear operator, A, the preconditioner P or my provided > restriction matrix RL1. A and P i'm pretty sure are OK, since it works just > fine when I use my usual BlockJacobi/Additive Schwartz with ILU(k) on the > subdomains. Which leaves the Restriction operator as the issue. I added the > following code before PCView() > > call MatGetVecs(RL1, vec1, vec2, ierr) > call VecSet(vec1, 1.0, ierr) > call MatMult(RL1, vec1, vec2, ierr) > > which ran through just fine (I check each of the ierrs but those are not > shown). > > So I'm at a loss as to what is causing the error. > I'm using an up-to-date copy of petsc-3.3 from the repository. > > Thanks, > > Gaetan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 31 20:52:31 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 31 Aug 2012 20:52:31 -0500 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: <504161F9.2070605@geology.wisc.edu> Message-ID: On Fri, Aug 31, 2012 at 8:45 PM, Mohammad Mirzadeh wrote: > Alright. it was not MKL either! > > Tabrez, I guess I could do that ... I just want to be sure this is a nasty > bug in compiler/mpi and/or undefined behavior in my code; those things are > quite nasty. Should I try a different MPI implementation? Right now its > mvapich2/1.6 > I am really sick of MVAPICH. We just spent almost two weeks on what turned out to be an MVAPICH bug. Quality control is almost entirely absent here. At least try MPICH or OpenMPI. Matt > On Fri, Aug 31, 2012 at 6:16 PM, Tabrez Ali wrote: > >> You can try the configure flags used by PETSc (3.2) that is >> pre-installed on Lonestar (configured with Hypre, MKL, Intel MPI etc.). >> >> $ module load petsc >> >> and then try any of the PETSc examples with bcgs hypre/none >> >> On 08/31/2012 07:46 PM, Mohammad Mirzadeh wrote: >> >> Alright, just removed pthreadclasses and it did not help. The code still >> crashes. >> >> Matt, I'm gonna try MKL now; However, don't you think its strange that >> the gcc version works even when I use MKL? >> >> On Fri, Aug 31, 2012 at 5:12 PM, Matthew Knepley wrote: >> >>> On Fri, Aug 31, 2012 at 6:51 PM, Jed Brown wrote: >>> >>>> Can you remove pthreadclasses from your configure? If the problem is >>>> still there, it looks a bit like a misconfigured MPI. But get rid of the >>>> pthreadclasses first. >>> >>> >>> After that, I would get rid of MKL in favor of >>> --download-f-blas-lapack. My guess is that MKL is doing something >>> crazy in the compiler. >>> >>> Matt >>> >>> >>>> On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh < >>>> mirzadeh at gmail.com> wrote: >>>> >>>>> hi guys, >>>>> >>>>> After doing many many tests across different machine (Linux box and >>>>> a Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in >>>>> which my code crashes if I use a petsc version that is compiled with intel >>>>> compiler. This does not happen if I use gcc. When I use BiCGSTAB solver >>>>> along with hypre's BoomerAMG, I get the following error message: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Error in external library! >>>>> [0]PETSC ERROR: Error in HYPRE solver, error code 1! >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>>> 15:42:00 CDT 2012 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>>> --download-superlu_dist=1 >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: PCApply_HYPRE() line 160 in >>>>> src/ksp/pc/impls/hypre/hypre.c >>>>> [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: KSPInitialResidual() line 57 in >>>>> src/ksp/ksp/interface/itres.c >>>>> [0]PETSC ERROR: KSPSolve_BCGS() line 65 in >>>>> src/ksp/ksp/impls/bcgs/bcgs.c >>>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: solve() line 402 in >>>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Error in external library! >>>>> [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>>> 15:42:00 CDT 2012 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>>> --download-superlu_dist=1 >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in >>>>> src/ksp/pc/impls/hypre/hypre.c >>>>> [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: clear() line 438 in >>>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>> probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>> -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >>>>> corruption errors >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>> available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c >>>>> [0]PETSC ERROR: [0] KSPReset_BCGS line 194 >>>>> src/ksp/ksp/impls/bcgs/bcgs.c >>>>> [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>>>> [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown >>>>> [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 >>>>> src/ksp/pc/impls/hypre/hypre.c >>>>> [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c >>>>> [0]PETSC ERROR: [0] PCApply_HYPRE line 149 >>>>> src/ksp/pc/impls/hypre/hypre.c >>>>> [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: [0] KSPInitialResidual line 45 >>>>> src/ksp/ksp/interface/itres.c >>>>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c >>>>> [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Signal received! >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>>> 15:42:00 CDT 2012 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>>> --download-superlu_dist=1 >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>>>> unknown file >>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>> >>>>> >>>>> Initially I thought this is somehow related to hypre, but even when >>>>> I run the code with -pc_type none I get: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Floating point exception! >>>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 >>>>> 15:42:00 CDT 2012 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./2D_cylinders on a arch-linu named >>>>> c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib >>>>> [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug >>>>> --with-clanguage=cxx --with-pthreadclasses=1 >>>>> --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 >>>>> --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 >>>>> --download-hdf5=1 --download-metis=1 --download-parmetis=1 >>>>> --download-superlu_dist=1 >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c >>>>> [0]PETSC ERROR: KSPSolve_BCGS() line 78 in >>>>> src/ksp/ksp/impls/bcgs/bcgs.c >>>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: solve() line 402 in >>>>> "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp >>>>> N5PETSc9ExceptionE >>>>> >>>>> >>>>> In both cases the solution to the linear system diverges. None of >>>>> these issues happen when I build petsc with gcc. >>>>> >>>>> Couple of points: >>>>> >>>>> 1) The gcc version is clean under valgrind (except the usual >>>>> PetscInitialize errors ...) but the intel versions generates all sort of >>>>> errors in valgrind. Please see blow (this is during one Poisson solve) >>>>> 2) This is my petsc configure option: >>>>> >>>>> /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx >>>>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>>>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>>>> >>>>> ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx >>>>> --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME >>>>> --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 >>>>> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 >>>>> >>>>> >>>>> in both cases $TACC_MKL_DIR points to a mkl library compiled using >>>>> intel compiler. I don't think if that has to do with any of this (looking >>>>> at valgrind output) but then again that's the only version of mkl available >>>>> on the cluster (Lonestar). >>>>> >>>>> Do you think this is somehow a compiler bug(?) or something more >>>>> like an undefined behavior in my own code? >>>>> >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x8283154: __intel_new_memset (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>>> ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x8283136: __intel_new_memset (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>>> ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe >>>>> (handlemem.c:363) >>>>> ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) >>>>> ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) >>>>> ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) >>>>> ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) >>>>> ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) >>>>> ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) >>>>> ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:331) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) >>>>> ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) >>>>> ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) >>>>> ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) >>>>> ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) >>>>> ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, >>>>> char const*, char const*) (petscLinearSolver.cpp:332) >>>>> ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() >>>>> (pnpSolver_2d_FVM.cpp:46) >>>>> ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) >>>>> ==8404== >>>>> Solving a nonlinear PB to get steady state solution ... >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR >>>>> (IJMatrix_parcsr.c:179) >>>>> ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes >>>>> (HYPRE_IJMatrix.c:818) >>>>> ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, >>>>> _p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:48) >>>>> ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, >>>>> hypre_IJMatrix_struct**) (mhyp.c:80) >>>>> ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== >>>>> ==8404== Syscall param mmap(length) contains uninitialised byte(s) >>>>> ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) >>>>> ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) >>>>> ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) >>>>> (bvec2.c:887) >>>>> ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) >>>>> ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) >>>>> ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) >>>>> (matrix.c:8149) >>>>> ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>>> (IJVector_parcsr.c:124) >>>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>>> (HYPRE_IJVector.c:229) >>>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>>> hypre_IJVector_struct**) (vhyp.c:21) >>>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>>> (IJVector_parcsr.c:124) >>>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>>> (HYPRE_IJVector.c:229) >>>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>>> hypre_IJVector_struct**) (vhyp.c:21) >>>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>>> ==8404== by 0x14E0E57: hypre_IJVectorInitializePar >>>>> (IJVector_parcsr.c:124) >>>>> ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize >>>>> (HYPRE_IJVector.c:229) >>>>> ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, >>>>> hypre_IJVector_struct**) (vhyp.c:21) >>>>> ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) >>>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>>> const*, char const*) (mal.c:75) >>>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>>> char const*, char const*) (mtr.c:322) >>>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) >>>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>>> const*, char const*) (mal.c:75) >>>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>>> char const*, char const*) (mtr.c:322) >>>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) >>>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>>> ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char >>>>> const*, char const*) (mal.c:75) >>>>> ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, >>>>> char const*, char const*) (mtr.c:322) >>>>> ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) >>>>> ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) >>>>> ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) >>>>> ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize >>>>> (par_csr_matrix.c:200) >>>>> ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:272) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize >>>>> (aux_parcsr_matrix.c:177) >>>>> ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR >>>>> (IJMatrix_parcsr.c:273) >>>>> ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize >>>>> (HYPRE_IJMatrix.c:303) >>>>> ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:211) >>>>> ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, >>>>> hypre_IJMatrix_struct*) (mhyp.c:122) >>>>> ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) >>>>> ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) >>>>> ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge >>>>> (par_coarsen.c:940) >>>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>>> (par_coarsen.c:1918) >>>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge >>>>> (par_coarsen.c:940) >>>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>>> (par_coarsen.c:1918) >>>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1012) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1012) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1012) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1122) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1122) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1122) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) >>>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>>>> ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1685) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== by 0x472A92: >>>>> amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) >>>>> (pbSolver_2d_FVM.cpp:148) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate >>>>> (par_csr_matrix.c:60) >>>>> ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) >>>>> ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x828313B: __intel_new_memset (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge >>>>> (par_coarsen.c:988) >>>>> ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout >>>>> (par_coarsen.c:1918) >>>>> ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x8283145: __intel_new_memset (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1354) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) >>>>> ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) >>>>> ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:372) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) >>>>> ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) >>>>> ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) >>>>> ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) >>>>> ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1767) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Use of uninitialised value of size 8 >>>>> ==8404== at 0x8283131: __intel_new_memset (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x8261535: _intel_fast_memset.J (in >>>>> /opt/apps/intel/11.1/lib/intel64/libirc.so) >>>>> ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) >>>>> ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) >>>>> ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator >>>>> (par_rap.c:1355) >>>>> ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) >>>>> ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) >>>>> ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) >>>>> ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) >>>>> ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) >>>>> ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:402) >>>>> ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, >>>>> PetscBool) (petscLinearSolver.cpp:402) >>>>> ==8404== >>>>> ==8404== Conditional jump or move depends on uninitialised value(s) >>>>> ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) >>>>> ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) >>>>> ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) >>>>> ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mal.c:30) >>>>> ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char >>>>> const*, char const*, char const*, void**) (mtr.c:190) >>>>> ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) >>>>> (vecstash.c:37) >>>>> ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, >>>>> int, double const*) (pbvec.c:207) >>>>> ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) >>>>> (pbvec.c:70) >>>>> ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) >>>>> ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, >>>>> KSPConvergedReason*, void*) (iterativ.c:585) >>>>> ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) >>>>> ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) >>>>> (itfunc.c:446) >>>>> ==8404== >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 31 21:27:48 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 21:27:48 -0500 Subject: [petsc-users] Petsc crashes with intel compiler In-Reply-To: References: Message-ID: <7EA29707-596C-4F49-AB71-3E477AEF7223@mcs.anl.gov> Try a different version of the Intel compiler; it is a good chance it is intel compiler bugs. Barry On Aug 31, 2012, at 6:53 PM, Mohammad Mirzadeh wrote: > I'll try it right away > > On Fri, Aug 31, 2012 at 4:51 PM, Jed Brown wrote: > Can you remove pthreadclasses from your configure? If the problem is still there, it looks a bit like a misconfigured MPI. But get rid of the pthreadclasses first. > > > On Fri, Aug 31, 2012 at 6:38 PM, Mohammad Mirzadeh wrote: > hi guys, > > After doing many many tests across different machine (Linux box and a Mac) and clusters (Gordon and Lonestar) I'm able to reproduce a test in which my code crashes if I use a petsc version that is compiled with intel compiler. This does not happen if I use gcc. When I use BiCGSTAB solver along with hypre's BoomerAMG, I get the following error message: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE solver, error code 1! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApply_HYPRE() line 160 in src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCApply() line 384 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPInitialResidual() line 57 in src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: KSPSolve_BCGS() line 65 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error in HYPRE_IJMatrixDestroy()! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCDestroy_HYPRE() line 178 in src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: PCDestroy() line 119 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPDestroy() line 786 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: clear() line 438 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecDestroy line 574 src/vec/vec/interface/vector.c > [0]PETSC ERROR: [0] KSPReset_BCGS line 194 src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPReset line 729 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] HYPRE_IJMatrixDestroy line 0 unknownunknown > [0]PETSC ERROR: [0] PCDestroy_HYPRE line 177 src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCDestroy line 110 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPDestroy line 768 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscError line 343 src/sys/error/err.c > [0]PETSC ERROR: [0] PCApply_HYPRE line 149 src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCApply line 373 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPInitialResidual line 45 src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_BCGS line 54 src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: [0] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:13:15 2012 > [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > Initially I thought this is somehow related to hypre, but even when I run the code with -pc_type none I get: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./2D_cylinders on a arch-linu named c341-116.ls4.tacc.utexas.edu by mmirzade Fri Aug 31 18:15:04 2012 > [0]PETSC ERROR: Libraries linked from /work/02032/mmirzade/soft/petsc-3.3-p2/arch-linux-intel-debug/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 17:22:42 2012 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: KSPSolve_BCGS() line 78 in src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve() line 402 in "unknowndirectory/"/work/02032/mmirzade/codes/CASL/lib/algebra/petscLinearSolver.cpp > N5PETSc9ExceptionE > > > In both cases the solution to the linear system diverges. None of these issues happen when I build petsc with gcc. > > Couple of points: > > 1) The gcc version is clean under valgrind (except the usual PetscInitialize errors ...) but the intel versions generates all sort of errors in valgrind. Please see blow (this is during one Poisson solve) > 2) This is my petsc configure option: > > /configure PETSC_ARCH=arch-linux-gcc-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > ./configure PETSC_ARCH=arch-linux-intel-debug --with-clanguage=cxx --with-pthreadclasses=1 --with-mpi-dir=$MPICH_HOME --with-blas-lapack-dir=$TACC_MKL_DIR --download-hypre=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > > in both cases $TACC_MKL_DIR points to a mkl library compiled using intel compiler. I don't think if that has to do with any of this (looking at valgrind output) but then again that's the only version of mkl available on the cluster (Lonestar). > > Do you think this is somehow a compiler bug(?) or something more like an undefined behavior in my own code? > > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283154: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D79D: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283136: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x747D714: MPIU_Handle_obj_alloc_unsafe (handlemem.c:363) > ==8404== by 0x747D665: MPIU_Handle_obj_alloc (handlemem.c:307) > ==8404== by 0x744F596: MPIR_Comm_create (commutil.c:101) > ==8404== by 0x744E84D: MPIR_Comm_copy (commutil.c:969) > ==8404== by 0x7447F14: PMPI_Comm_dup (comm_dup.c:177) > ==8404== by 0xB2292A: PCCreate_HYPRE (hypre.c:1018) > ==8404== by 0xA64120: PCSetType(_p_PC*, char const*) (pcset.c:83) > ==8404== by 0x46B3FE: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:331) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B236A: calloc (mvapich_malloc.c:3756) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14E6385: hypre_BoomerAMGCreate (par_amg.c:203) > ==8404== by 0x14E2E10: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31) > ==8404== by 0xB1C33B: PCHYPRESetType_HYPRE (hypre.c:817) > ==8404== by 0xB2117F: PCSetFromOptions_HYPRE(_p_PC*) (hypre.c:895) > ==8404== by 0xA65705: PCSetFromOptions(_p_PC*) (pcset.c:196) > ==8404== by 0x46B487: PetscLinearSolver::setupSolver(MatStructure, char const*, char const*) (petscLinearSolver.cpp:332) > ==8404== by 0x46E7D1: amr2D::fvm::pnpSolver::initialize() (pnpSolver_2d_FVM.cpp:46) > ==8404== by 0x40D464: main (main_2D_cylinders.cpp:299) > ==8404== > Solving a nonlinear PB to get steady state solution ... > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74B21CD: calloc (mvapich_malloc.c:3825) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14DB097: hypre_IJMatrixSetDiagOffdSizesParCSR (IJMatrix_parcsr.c:179) > ==8404== by 0x14D97C9: HYPRE_IJMatrixSetDiagOffdSizes (HYPRE_IJMatrix.c:818) > ==8404== by 0x11A2728: MatHYPRE_IJMatrixPreallocate(_p_Mat*, _p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:48) > ==8404== by 0x11A3E9C: MatHYPRE_IJMatrixCreate(_p_Mat*, hypre_IJMatrix_struct**) (mhyp.c:80) > ==8404== by 0xAFDA92: PCSetUp_HYPRE(_p_PC*) (hypre.c:100) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AECA4: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Syscall param mmap(length) contains uninitialised byte(s) > ==8404== at 0x3E3CED10EA: mmap (in /lib64/libc-2.5.so) > ==8404== by 0x74AECD1: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA3E: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA70: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AEA8A: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AEA98: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD6E6: _int_free (mvapich_malloc.c:4366) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD723: _int_free (mvapich_malloc.c:4389) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD755: _int_free (mvapich_malloc.c:4400) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AD7B4: _int_free (mvapich_malloc.c:4455) > ==8404== by 0x74AEAA8: _int_malloc (mvapich_malloc.c:4332) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5FA4BA: VecCreate_Seq (bvec3.c:40) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0x5E9616: VecCreate_Seq_Private(_p_Vec*, double const*) (bvec2.c:887) > ==8404== by 0x5FA62A: VecCreate_Seq (bvec3.c:42) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x614EA3: VecCreate_Standard (pbvec.c:264) > ==8404== by 0x5B36DC: VecSetType(_p_Vec*, char const*) (vecreg.c:53) > ==8404== by 0x703809: MatGetVecs(_p_Mat*, _p_Vec**, _p_Vec**) (matrix.c:8149) > ==8404== by 0xAFDB5E: PCSetUp_HYPRE(_p_PC*) (hypre.c:104) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADEE6: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF31: _int_malloc (mvapich_malloc.c:4216) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14E0E57: hypre_IJVectorInitializePar (IJVector_parcsr.c:124) > ==8404== by 0x14DA2B9: HYPRE_IJVectorInitialize (HYPRE_IJVector.c:229) > ==8404== by 0xE77A78: VecHYPRE_IJVectorCreate(_p_Vec*, hypre_IJVector_struct**) (vhyp.c:21) > ==8404== by 0xAFDC0C: PCSetUp_HYPRE(_p_PC*) (hypre.c:105) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD75B: _int_free (mvapich_malloc.c:4402) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD775: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD7A9: _int_free (mvapich_malloc.c:4425) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x494CCA: PetscFreeAlign(void*, int, char const*, char const*, char const*) (mal.c:75) > ==8404== by 0x497F91: PetscTrFreeDefault(void*, int, char const*, char const*, char const*) (mtr.c:322) > ==8404== by 0x5E866E: VecDestroy_Seq(_p_Vec*) (bvec2.c:777) > ==8404== by 0x59F7CD: VecDestroy(_p_Vec**) (vector.c:580) > ==8404== by 0xAFDD5D: PCSetUp_HYPRE(_p_PC*) (hypre.c:107) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDA1: _int_malloc (mvapich_malloc.c:4106) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDAE: _int_malloc (mvapich_malloc.c:4116) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDD9: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDEC: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADDFF: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE12: _int_malloc (mvapich_malloc.c:4122) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE42: _int_malloc (mvapich_malloc.c:4124) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE4C: _int_malloc (mvapich_malloc.c:4126) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE88: _int_malloc (mvapich_malloc.c:4148) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE9C: _int_malloc (mvapich_malloc.c:4152) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3AA1: hypre_CSRMatrixInitialize (csr_matrix.c:91) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1E7: _int_malloc (mvapich_malloc.c:4170) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AE1EF: _int_malloc (mvapich_malloc.c:4173) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE266: _int_malloc (mvapich_malloc.c:4188) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3B06: hypre_CSRMatrixInitialize (csr_matrix.c:97) > ==8404== by 0x158732D: hypre_ParCSRMatrixInitialize (par_csr_matrix.c:200) > ==8404== by 0x14DB366: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:272) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF87: _int_malloc (mvapich_malloc.c:4237) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1626536: hypre_AuxParCSRMatrixInitialize (aux_parcsr_matrix.c:177) > ==8404== by 0x14DB372: hypre_IJMatrixInitializeParCSR (IJMatrix_parcsr.c:273) > ==8404== by 0x14D8D44: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:303) > ==8404== by 0x11A7FFF: MatHYPRE_IJMatrixFastCopy_MPIAIJ(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:211) > ==8404== by 0x11A4BFD: MatHYPRE_IJMatrixCopy(_p_Mat*, hypre_IJMatrix_struct*) (mhyp.c:122) > ==8404== by 0xAFE3DC: PCSetUp_HYPRE(_p_PC*) (hypre.c:118) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADECC: _int_malloc (mvapich_malloc.c:4165) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B62C5: hypre_SeqVectorInitialize (vector.c:94) > ==8404== by 0x158CF25: hypre_ParVectorInitialize (par_vector.c:150) > ==8404== by 0x14ECF47: hypre_BoomerAMGSetup (par_amg_setup.c:464) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF72: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF7E: _int_malloc (mvapich_malloc.c:4227) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC3F9: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:940) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE293: _int_malloc (mvapich_malloc.c:4254) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74AE29B: _int_malloc (mvapich_malloc.c:4257) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE327: _int_malloc (mvapich_malloc.c:4278) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153D557: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1012) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE4E: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADE56: _int_malloc (mvapich_malloc.c:4129) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74ADE98: _int_malloc (mvapich_malloc.c:4151) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153DAD9: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1122) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD771: _int_free (mvapich_malloc.c:4406) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x153F2EA: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1685) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== by 0x472A92: amr2D::fvm::pbSolver::SolveNonLinearPB(double, int) (pbSolver_2d_FVM.cpp:148) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADF56: _int_malloc (mvapich_malloc.c:4220) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x1586F83: hypre_ParCSRMatrixCreate (par_csr_matrix.c:60) > ==8404== by 0x1554BE0: hypre_BoomerAMGCreateS (par_strength.c:156) > ==8404== by 0x14EDD1B: hypre_BoomerAMGSetup (par_amg_setup.c:736) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x828313B: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x14FC57D: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:988) > ==8404== by 0x14FF1EE: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1918) > ==8404== by 0x14EDE45: hypre_BoomerAMGSetup (par_amg_setup.c:758) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283145: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2B7: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1354) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AE1F1: _int_malloc (mvapich_malloc.c:4174) > ==8404== by 0x74B2189: calloc (mvapich_malloc.c:3765) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x15B3457: hypre_CSRMatrixTranspose (csr_matop.c:354) > ==8404== by 0x153B7B8: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:372) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x74AD77E: _int_free (mvapich_malloc.c:4409) > ==8404== by 0x74AF651: free (mvapich_malloc.c:3497) > ==8404== by 0x16240BA: hypre_Free (hypre_memory.c:196) > ==8404== by 0x15B3A2F: hypre_CSRMatrixDestroy (csr_matrix.c:69) > ==8404== by 0x153F6C2: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1767) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Use of uninitialised value of size 8 > ==8404== at 0x8283131: __intel_new_memset (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x8261535: _intel_fast_memset.J (in /opt/apps/intel/11.1/lib/intel64/libirc.so) > ==8404== by 0x74B221A: calloc (mvapich_malloc.c:3826) > ==8404== by 0x1623FF2: hypre_CAlloc (hypre_memory.c:121) > ==8404== by 0x153E2DD: hypre_BoomerAMGBuildCoarseOperator (par_rap.c:1355) > ==8404== by 0x14F1CCB: hypre_BoomerAMGSetup (par_amg_setup.c:1650) > ==8404== by 0x14E2EA6: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58) > ==8404== by 0xAFF31D: PCSetUp_HYPRE(_p_PC*) (hypre.c:122) > ==8404== by 0x12293CE: PCSetUp(_p_PC*) (precon.c:832) > ==8404== by 0xC1D524: KSPSetUp(_p_KSP*) (itfunc.c:278) > ==8404== by 0xC1EFD9: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:402) > ==8404== by 0x46C59F: PetscLinearSolver::solve(ArrayV&, PetscBool) (petscLinearSolver.cpp:402) > ==8404== > ==8404== Conditional jump or move depends on uninitialised value(s) > ==8404== at 0x74ADED9: _int_malloc (mvapich_malloc.c:4168) > ==8404== by 0x74AEE8D: malloc (mvapich_malloc.c:3409) > ==8404== by 0x74AFB0F: memalign (mvapich_malloc.c:3626) > ==8404== by 0x494BF3: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*, void**) (mal.c:30) > ==8404== by 0x496DFD: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char const*, void**) (mtr.c:190) > ==8404== by 0xE71ECC: VecStashCreate_Private(int, int, VecStash*) (vecstash.c:37) > ==8404== by 0x6145A1: VecCreate_MPI_Private(_p_Vec*, PetscBool, int, double const*) (pbvec.c:207) > ==8404== by 0x613475: VecDuplicate_MPI(_p_Vec*, _p_Vec**) (pbvec.c:70) > ==8404== by 0x59EF10: VecDuplicate(_p_Vec*, _p_Vec**) (vector.c:551) > ==8404== by 0xC4F687: KSPDefaultConverged(_p_KSP*, int, double, KSPConvergedReason*, void*) (iterativ.c:585) > ==8404== by 0xC6CB92: KSPSolve_BCGS(_p_KSP*) (bcgs.c:86) > ==8404== by 0xC1FC32: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:446) > ==8404== > > > > > > From bsmith at mcs.anl.gov Fri Aug 31 21:37:13 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 21:37:13 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: <74097C65-5C65-488E-840E-91B825A5F93E@mcs.anl.gov> You should try running your code with valgrind. It could be the memory corruption is early and just happens to appear here. Barry On Aug 31, 2012, at 8:48 PM, Jed Brown wrote: > src/ksp/pc/examples/tests/ex8f.F is a simple example, but I made the modification below and it runs fine. > > diff --git a/src/ksp/pc/examples/tests/ex8f.F b/src/ksp/pc/examples/tests/ex8f.F > --- a/src/ksp/pc/examples/tests/ex8f.F > +++ b/src/ksp/pc/examples/tests/ex8f.F > @@ -35,7 +35,7 @@ > Vec x,b,u > PC pc > PetscInt n,dim,istart,iend > - PetscInt i,j,jj,ii,one,zero > + PetscInt i,j,jj,ii,one,two,zero > PetscErrorCode ierr > PetscScalar v > external MyResidual > @@ -50,6 +50,7 @@ > pfive = .5d0 > n = 6 > dim = n*n > + two = 2 > one = 1 > zero = 0 > > @@ -138,7 +139,7 @@ > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCMG,ierr) > - call PCMGSetLevels(pc,one,PETSC_NULL_OBJECT,ierr) > + call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) > call PCMGSetResidual(pc,zero,PCMGDefaultResidual,A,ierr) > > call PCMGSetResidual(pc,zero,MyResidual,A,ierr) > @@ -147,6 +148,7 @@ > ! also serves as the preconditioning matrix. > > call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr) > + call KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD,ierr) > > > call KSPDestroy(ksp,ierr) > > $ mpirun.hydra -n 2 ./ex8f > KSP Object: 2 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 2 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 2 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 2 MPI processes > type: redundant > Redundant preconditioner: Not yet setup > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 2 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0, max = 0 > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 2 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 2 MPI processes > type: mpiaij > rows=36, cols=36 > total: nonzeros=156, allocated nonzeros=252 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > On Fri, Aug 31, 2012 at 7:27 PM, Gaetan Kenway wrote: > Hi > > I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to use PCMG to precondition a linear system of equations resulting from the discretization of the EUler equations. It is from a multi-block code so I have the ability to easily generate restriction operators using geometric method. > > I'm currently trying to setup a simple 2V cycle. For this I use: > > ! Create the restriction operator between finest level and one below > call createRestrictOperator(RL1, 1) > > call PCSetFromOptions(pc, PETScIerr) > nlevels=2 > !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't work from fortran!!! > call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr) > call PCMGSetRestriction(pc, 1, RL1, PETScierr) > > call PCView(pc, PETScierr) > > I run with the command line argument -pc_mg_levels 2 . > > First issue: PCMGSetLevels does not work correctly from Fortran. The traceback I get when I run (with the commented-out line above and without the command line arguments) is: > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscCommDuplicate line 144 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c > [0]PETSC ERROR: [0] KSPCreate line 557 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c > [0]PETSC ERROR: [0] PCMGSetLevels line 180 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c > > I can get around that issue by using -pc_mg_levels 2 and not calling the PCMGSetLevels() command, but I would like to actually use this command and not have set it via manually setting an option. The result of the PCView is listed below which appears to be what I expect (given the default options) > > PC Object: 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 1 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 1 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0, max = 0 > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix followed by preconditioner matrix: > Matrix Object: 1 MPI processes > type: seqbaij > rows=92160, cols=92160, bs=5 > total: nonzeros=5582400, allocated nonzeros=5582400 > total number of mallocs used during MatSetValues calls =0 > block size is 5 > Matrix Object: 1 MPI processes > type: seqbaij > rows=92160, cols=92160, bs=5 > total: nonzeros=3089600, allocated nonzeros=3089600 > total number of mallocs used during MatSetValues calls =0 > block size is 5 > > > So that's ok. When it gets to actually solving my system with KSPSolve() I get a returned error code 73 which is "object in argument is in wrong state, e.g. unassembled mat" > > In general, please don't paraphrase error messages. > > With these problems, try running in a debugger to get a real trace and with valgrind. The most common problem is stack corruption due to passing the wrong number or types of arguments. > > > and the traceback is: > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatGetVecs line 8142 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] KSPGetVecs line 774 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: [0] PCSetUp_MG line 508 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCSetUp line 810 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 182 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 351 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > > Which would indicate to me its an issue with one of the matrices involved, either the linear operator, A, the preconditioner P or my provided restriction matrix RL1. A and P i'm pretty sure are OK, since it works just fine when I use my usual BlockJacobi/Additive Schwartz with ILU(k) on the subdomains. Which leaves the Restriction operator as the issue. I added the following code before PCView() > > call MatGetVecs(RL1, vec1, vec2, ierr) > call VecSet(vec1, 1.0, ierr) > call MatMult(RL1, vec1, vec2, ierr) > > which ran through just fine (I check each of the ierrs but those are not shown). > > So I'm at a loss as to what is causing the error. > I'm using an up-to-date copy of petsc-3.3 from the repository. > > Thanks, > > Gaetan > From bsmith at mcs.anl.gov Fri Aug 31 21:57:58 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 31 Aug 2012 21:57:58 -0500 Subject: [petsc-users] pctype hmpi In-Reply-To: References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> <2EABC8B5-DF3A-4B05-A49E-E26566C31979@mcs.anl.gov> Message-ID: <8883F05E-51DE-4B37-82BB-102A3925505E@mcs.anl.gov> Ok, another bug. Put the attached file in src/mat/impls/aij/mpi and run make in that directory then relink program and run again. Barry [see attached file: mpiaij.c] On Aug 31, 2012, at 5:26 PM, George Pau wrote: > hi Barry, > > The hmpi option is read in properly now. The error is now different when I use -hmpi_spawn_size 3 with mpiexec -n 1. My printout suggests this is now happening in the KSPSolve. > > George > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [0] petscinitialize_(): Running on machine: gilbert > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [0] petscinitialize_(): Running on machine: gilbert > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > [1] petscinitialize_(): Running on machine: gilbert > [0] PetscHMPISpawn(): PETSc HMPI successfully spawned: number of nodes = 1 node size = 3 > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374781 max tags = 2147483647 > [0] MatSetUp(): Warning not preallocating matrix storage > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781 > start ksp > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781 > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Duplicating a communicator -2080374782 -2080374780 max tags = 2147483647 > [0] PetscCommDuplicate(): Duplicating a communicator -1006632960 -1006632959 max tags = 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator -2080374779 -2080374778 max tags = 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 -2080374780 > [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 -1006632959 > [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 -2080374778 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374779 max tags = 2147483647 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -1006632958 max tags = 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374779 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -1006632958 > [0] VecScatterCreate(): Special case: processor zero gets entire parallel vector, rest get none > [0] Petsc_DelComm(): Removing reference to PETSc communicator imbedded in a user MPI_Comm m -2080374779 > [0] Petsc_DelComm(): User MPI_Comm m 1140850689 is being freed, removing reference from inner PETSc comm to this outer comm > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374779 > [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374779 > [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 -2080374780 > [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 -1006632959 > [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 -2080374778 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -1006632958 > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] MatDistribute_MPIAIJ line 192 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: [1] PCSetUp_HMPI_MP line 90 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > [1]PETSC ERROR: [1] PetscHMPIHandle line 253 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > [1]PETSC ERROR: [1] PetscHMPISpawn line 71 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 15:20:27 2012 > [1]PETSC ERROR: [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > [1]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 > [1]PETSC ERROR: Configure options --with-debugging=1 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > [0]PETSC ERROR: [0] MatDistribute_MPIAIJ line 192 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: [0] PCSetUp_HMPI_MP line 90 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > [0]PETSC ERROR: [0] PetscHMPIHandle line 253 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > [0]PETSC ERROR: [0] PetscHMPISpawn line 71 /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 15:20:27 2012 > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > [0]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 > [0]PETSC ERROR: Configure options --with-debugging=1 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > Fatal error in MPI_Allreduce: Other MPI error, error stack: > MPI_Allreduce(855)........: MPI_Allreduce(sbuf=0x7fff6315ead0, rbuf=0x7fff6315eae0, count=2, MPI_INT, MPI_MAX, comm=0x84000004) failed > MPIR_Allreduce_impl(712)..: > MPIR_Allreduce_intra(534).: > dequeue_and_set_error(596): Communication error with rank 0 > > > On Fri, Aug 31, 2012 at 2:09 PM, Barry Smith wrote: > [see attached file: zstart.c] > > On Aug 31, 2012, at 4:07 PM, George Pau wrote: > > > Hi Barry, > > > > You forgot the file ... > > > > George > > > > On Fri, Aug 31, 2012 at 2:04 PM, Barry Smith wrote: > > > > Yikes. It is totally my fault. The handling of these merge and spawn options is done only for PetscInitialize() for C. Not for Fortran, hence the arguments just got ignored. > > > > Please find attached a file zstart.c put it in the directory src/sys/ftn-custom and run make in that directory (with appropriate PETSC_DIR and PETSC_ARCH set) > > > > Then link and run the example again. > > > > > > Barry > > > > > > On Aug 31, 2012, at 3:30 PM, George Pau wrote: > > > > > Sorry, it was a cut and paste error. I tried running the code with all the options in the command line: > > > > > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre boomeramg > > > > > > but I get the exact same outputs. > > > > > > George > > > > > > > > > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith wrote: > > > > > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > > > > > Hi Barry, > > > > > > > > For the hmpi_spawn_size, the options in my .petscrc are > > > > -info > > > > -pc_view > > > > pc_type hmpi > > > > > > How come there is no - in front of this one? > > > > > > > -ksp_type preonly > > > > -ksp_view > > > > -hmpi_pc_monitor > > > > -hmpi_ksp_monitor > > > > -hmpi_ksp_type cg > > > > -hmpi_pc_type hypre > > > > -hmpi_pc_hypre_type boomeramg > > > > -hmpi_spawn_size 3 > > > > > > > > mpiexec.mpich2 -n 1 myprogram > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 unneeded,3222 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 360 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode routines > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 > > > > > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, root=0, comm=0x0) failed > > > > PMPI_Bcast(1418): Invalid communicator > > > > > > > > I inserted some print statement between the ksp calls and found that the error occurs in > > > > > > > > call KSPSetFromOptions(ksp, pierr) > > > > > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch my job by > > > > > > How come there is no - in front of hmpi_merge_size 2? > > > > > > > > > Can you try putting all the arguments as command line arguments instead of in a file? It shouldn't matter but it seems like some of the arguments are being ignored. > > > > > > Barry > > > > > > > > > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS gilbert.(none) > > > > [1] petscinitialize_(): Running on machine: gilbert > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374780 max tags = 2147483647 > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374782 max tags = 2147483647 > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374777 max tags = 2147483647 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 -2080374780 max tags = 2147483647 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 mallocs. > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage space: 1998 unneeded,1602 used > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 180 > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using Inode routines > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374777 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780 > > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > > > [0] VecScatterCreate(): General case: MPI to Seq > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: 396 unneeded,9 used > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 3 > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374782 > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > > [0] PCSetUp(): Setting up new PC > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > > > > > > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > > > [0]PETSC ERROR: Nonconforming object sizes! > > > > [0]PETSC ERROR: HMPI preconditioner only works for sequential solves! > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > > > [0]PETSC ERROR: PCSetType() line 83 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > [0]PETSC ERROR: PCSetFromOptions() line 188 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > > > [0]PETSC ERROR: No support for this operation for this object type! > > > > [0]PETSC ERROR: PC does not have apply! > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > [0]PETSC ERROR: Libraries linked from /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: PCApply() line 382 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > > > [0]PETSC ERROR: KSPInitialResidual() line 64 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > > > [0]PETSC ERROR: KSPSolve() line 446 in /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > > > > > I note that the error appears to occur at the same point. > > > > > > > > George > > > > > > > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith wrote: > > > > > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and MPI_Bcast. > > > > > > > > That should not happen. Please run and send all the output including the exact command line used > > > > > > > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu but if MPICH version that Petsc download with -download-mpich works, I can use that. > > > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with more than 1 cpus. But, petsc will complain that the pctype hmpi can only be used in serial. > > > > > > > > That should not happen. Run with 2 MPI processes and -hmpi_merge_size 2 and send the complete error message. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > > > > > >> > > > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > > > >> > > > > >>> Hi Barry, > > > > >>> > > > > >>> I tried with the addition of > > > > >>> > > > > >>> -hmpi_spawn_size 3 > > > > >>> > > > > >>> but I am still getting the same error though. > > > > >> > > > > >> The EXACT same error? Or some other error? > > > > >> > > > > >> What happens if you run with the -hmpi_merge_size option instead? > > > > >> > > > > >> Barry > > > > >> > > > > >> 1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn. > > > > >> > > > > >> 2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off. > > > > >> > > > > >> > > > > >>> I am using mpich2. Any other options to try? > > > > >>> > > > > >>> George > > > > >>> > > > > >>> > > > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > > > >>> > > > > >>>> > > > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > > > >>>> > > > > >>>>> Hi, > > > > >>>>> > > > > >>>>> I have some issues using the -pctype hmpi. I used the same setting found at > > > > >>>>> > > > > >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > > > >>>>> > > > > >>>>> i.e. > > > > >>>>> -pc_type hmpi > > > > >>>>> -ksp_type preonly > > > > >>>>> -hmpi_ksp_type cg > > > > >>>>> -hmpi_pc_type hypre > > > > >>>>> -hmpi_pc_hypre_type boomeramg > > > > >>>>> > > > > >>>>> My command is > > > > >>>>> > > > > >>>>> mpiexec -n 1 myprogram > > > > >>>> > > > > >>>> Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has > > > > >>>> > > > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner > > > > >>>> > > > > >>>> This will tell you want option to start PETSc up with. > > > > >>>> > > > > >>>> I will fix the PC so that it prints a far more useful error message. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> Barry > > > > >>>> > > > > >>>> > > > > >>>>> > > > > >>>>> But, I get > > > > >>>>> > > > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > > > >>>>> > > > > >>>>> with openmpi. I get similar error with mpich2 > > > > >>>>> > > > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed > > > > >>>>> PMPI_Bcast(1418): Invalid communicator > > > > >>>>> > > > > >>>>> I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine. > > > > >>>>> > > > > >>>>> Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly? > > > > >>>>> > > > > >>>>> -- > > > > >>>>> George Pau > > > > >>>>> Earth Sciences Division > > > > >>>>> Lawrence Berkeley National Laboratory > > > > >>>>> One Cyclotron, MS 74-120 > > > > >>>>> Berkeley, CA 94720 > > > > >>>>> > > > > >>>>> (510) 486-7196 > > > > >>>>> gpau at lbl.gov > > > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > George Pau > > > > Earth Sciences Division > > > > Lawrence Berkeley National Laboratory > > > > One Cyclotron, MS 74-120 > > > > Berkeley, CA 94720 > > > > > > > > (510) 486-7196 > > > > gpau at lbl.gov > > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > > > > > > > -- > > > George Pau > > > Earth Sciences Division > > > Lawrence Berkeley National Laboratory > > > One Cyclotron, MS 74-120 > > > Berkeley, CA 94720 > > > > > > (510) 486-7196 > > > gpau at lbl.gov > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > -------------- next part -------------- A non-text attachment was scrubbed... Name: mpiaij.c Type: application/octet-stream Size: 225449 bytes Desc: not available URL: From kenway at utias.utoronto.ca Fri Aug 31 22:03:54 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Fri, 31 Aug 2012 23:03:54 -0400 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: Hi again I distilled the problem down further to a simple subroutine in mgtest.F, copied directly from ex8f.F subroutine MGTest() implicit none #include "include/finclude/petsc.h" KSP ksp PC pc PetscErrorCode ierr PetscInt nlevels, two two = 2 call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc,PCMG,ierr) call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) call KSPDestroy(ksp) end subroutine MGTest When I ran this simple case with gdb I got the following backtrace: #0 0xb694d465 in PMPI_Attr_get () from /usr/local/lib/libmpi.so.0 #1 0xb555d7c9 in PetscCommDuplicate (comm_in=0x0, comm_out=0x88d7378, first_tag=0x88d73a0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c:145 #2 0xb5562ef7 in PetscHeaderCreate_Private (h=0x88d7370, classid=1211235, type=-1, class_name=0xb61cfd70 "KSP", descr=0xb61cfd74 "Krylov Method", mansec=0xb61cfd70 "KSP", comm=0x0, des=0xb5eeac70 , vie=0xb5efafd7 ) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c:51 #3 0xb5eff7ca in KSPCreate (comm=0x0, inksp=0x88d6ee0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c:564 #4 0xb5d4ac3a in PCMGSetLevels (pc=0x89089a0, levels=2, comms=0xb4bbf5a0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c:219 #5 0xb5d2c73c in pcmgsetlevels_ (pc=0xbfffec98, levels=0xbfffec8c, comms=0xb4bbf5a0, ierr=0xbfffec94) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/ftn-custom/zmgf.c:15 #6 0xb4affe1d in mgtest () at mgtest.F:15 #7 0xb4aad769 in f2py_rout_warp_mgtest (capi_self=0x8618758, capi_args=0xb7c3002c, capi_keywds=0x0, f2py_func=0xb4affdc0 ) at warpmodule.c:4273 #8 0xb4aa58cf in fortran_call (fp=0x8618758, arg=0xb7c3002c, kw=0x0) at /usr/lib/python2.6/dist-packages/numpy/f2py/src/fortranobject.c:318 #9 0x0805fd6a in PyObject_Call () #10 0x080dd5b0 in PyEval_EvalFrameEx () #11 0x080dfbb2 in PyEval_EvalCodeEx () #12 0x080dfca7 in PyEval_EvalCode () #13 0x080fd956 in PyRun_FileExFlags () #14 0x080fdbb2 in PyRun_SimpleFileExFlags () #15 0x0805b6d3 in Py_Main () #16 0x0805a8ab in main () The only difference is that I'm calling the routine from python and petsc is initialized from petsc4py. Any ideas? Thanks, Gaetan On Fri, Aug 31, 2012 at 9:48 PM, Jed Brown wrote: > src/ksp/pc/examples/tests/ex8f.F is a simple example, but I made the > modification below and it runs fine. > > diff --git a/src/ksp/pc/examples/tests/ex8f.F > b/src/ksp/pc/examples/tests/ex8f.F > --- a/src/ksp/pc/examples/tests/ex8f.F > +++ b/src/ksp/pc/examples/tests/ex8f.F > @@ -35,7 +35,7 @@ > Vec x,b,u > PC pc > PetscInt n,dim,istart,iend > - PetscInt i,j,jj,ii,one,zero > + PetscInt i,j,jj,ii,one,two,zero > PetscErrorCode ierr > PetscScalar v > external MyResidual > @@ -50,6 +50,7 @@ > pfive = .5d0 > n = 6 > dim = n*n > + two = 2 > one = 1 > zero = 0 > > @@ -138,7 +139,7 @@ > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCMG,ierr) > - call PCMGSetLevels(pc,one,PETSC_NULL_OBJECT,ierr) > + call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) > call PCMGSetResidual(pc,zero,PCMGDefaultResidual,A,ierr) > > call PCMGSetResidual(pc,zero,MyResidual,A,ierr) > @@ -147,6 +148,7 @@ > ! also serves as the preconditioning matrix. > > call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr) > + call KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD,ierr) > > > call KSPDestroy(ksp,ierr) > > $ mpirun.hydra -n 2 ./ex8f > KSP Object: 2 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 2 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 2 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 2 MPI processes > type: redundant > Redundant preconditioner: Not yet setup > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 2 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0, max = 0 > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 2 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 2 MPI processes > type: mpiaij > rows=36, cols=36 > total: nonzeros=156, allocated nonzeros=252 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > On Fri, Aug 31, 2012 at 7:27 PM, Gaetan Kenway wrote: > >> Hi >> >> I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to >> use PCMG to precondition a linear system of equations resulting from the >> discretization of the EUler equations. It is from a multi-block code so I >> have the ability to easily generate restriction operators using geometric >> method. >> >> I'm currently trying to setup a simple 2V cycle. For this I use: >> >> ! Create the restriction operator between finest level and one below >> call createRestrictOperator(RL1, 1) >> >> call PCSetFromOptions(pc, PETScIerr) >> nlevels=2 >> !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't >> work from fortran!!! >> call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr) >> call PCMGSetRestriction(pc, 1, RL1, PETScierr) >> >> call PCView(pc, PETScierr) >> >> I run with the command line argument -pc_mg_levels 2 . >> >> First issue: PCMGSetLevels does not work correctly from Fortran. The >> traceback I get when I run (with the commented-out line above and without >> the command line arguments) is: >> >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PetscCommDuplicate line 144 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c >> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c >> [0]PETSC ERROR: [0] KSPCreate line 557 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c >> [0]PETSC ERROR: [0] PCMGSetLevels line 180 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c >> >> I can get around that issue by using -pc_mg_levels 2 and not calling the >> PCMGSetLevels() command, but I would like to actually use this command and >> not have set it via manually setting an option. The result of the PCView is >> listed below which appears to be what I expect (given the default options) >> >> PC Object: 1 MPI processes >> type: mg >> MG: type is MULTIPLICATIVE, levels=2 cycles=v >> Cycles per PCApply=1 >> Using Galerkin computed coarse grid matrices >> Coarse grid solver -- level ------------------------------- >> KSP Object: (mg_coarse_) 1 MPI processes >> type: preonly >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (mg_coarse_) 1 MPI processes >> type: lu >> LU: out-of-place factorization >> tolerance for zero pivot 2.22045e-14 >> matrix ordering: nd >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (mg_levels_1_) 1 MPI processes >> type: chebyshev >> Chebyshev: eigenvalue estimates: min = 0, max = 0 >> maximum iterations=2, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (mg_levels_1_) 1 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix followed by preconditioner matrix: >> Matrix Object: 1 MPI processes >> type: seqbaij >> rows=92160, cols=92160, bs=5 >> total: nonzeros=5582400, allocated nonzeros=5582400 >> total number of mallocs used during MatSetValues calls =0 >> block size is 5 >> Matrix Object: 1 MPI processes >> type: seqbaij >> rows=92160, cols=92160, bs=5 >> total: nonzeros=3089600, allocated nonzeros=3089600 >> total number of mallocs used during MatSetValues calls =0 >> block size is 5 >> >> >> So that's ok. When it gets to actually solving my system with KSPSolve() >> I get a returned error code 73 which is "object in argument is in wrong >> state, e.g. unassembled mat" >> > > In general, please don't paraphrase error messages. > > With these problems, try running in a debugger to get a real trace and > with valgrind. The most common problem is stack corruption due to passing > the wrong number or types of arguments. > > >> >> and the traceback is: >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] MatGetVecs line 8142 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: [0] KSPGetVecs line 774 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c >> [0]PETSC ERROR: [0] PCSetUp_MG line 508 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c >> [0]PETSC ERROR: [0] PCSetUp line 810 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPSetUp line 182 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] KSPSolve line 351 >> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c >> >> Which would indicate to me its an issue with one of the matrices >> involved, either the linear operator, A, the preconditioner P or my >> provided restriction matrix RL1. A and P i'm pretty sure are OK, since it >> works just fine when I use my usual BlockJacobi/Additive Schwartz with >> ILU(k) on the subdomains. Which leaves the Restriction operator as the >> issue. I added the following code before PCView() >> >> call MatGetVecs(RL1, vec1, vec2, ierr) >> call VecSet(vec1, 1.0, ierr) >> call MatMult(RL1, vec1, vec2, ierr) >> >> which ran through just fine (I check each of the ierrs but those are not >> shown). >> >> So I'm at a loss as to what is causing the error. >> I'm using an up-to-date copy of petsc-3.3 from the repository. >> >> Thanks, >> >> Gaetan >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 31 22:05:39 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 22:05:39 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 10:03 PM, Gaetan Kenway wrote: > I distilled the problem down further to a simple subroutine in mgtest.F, > copied directly from ex8f.F > > subroutine MGTest() > implicit none > > #include "include/finclude/petsc.h" > > KSP ksp > PC pc > PetscErrorCode ierr > PetscInt nlevels, two > two = 2 > > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCMG,ierr) > call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) > > call KSPDestroy(ksp) > You are certainly missing an argument (ierr) to KSPDestroy... -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Fri Aug 31 22:06:14 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Fri, 31 Aug 2012 23:06:14 -0400 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: Running the simple code through valgrind yields the only non-python related error below: ==23697== Invalid read of size 4 ==23697== at 0x6592465: PMPI_Attr_get (in /usr/local/lib/libmpi.so.0.0.2) ==23697== by 0x8BED7C8: PetscCommDuplicate (tagm.c:145) ==23697== by 0x8BF2EF6: PetscHeaderCreate_Private (inherit.c:51) ==23697== by 0x958F7C9: KSPCreate (itcreate.c:564) ==23697== by 0x93DAC39: PCMGSetLevels (mg.c:219) ==23697== by 0x93BC73B: pcmgsetlevels_ (zmgf.c:15) ==23697== by 0x7A5CE1C: mgtest_ (mgtest.F:15) ==23697== Address 0x8c is not stack'd, malloc'd or (recently) free'd ==23697== [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 144 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c [0]PETSC ERROR: [0] KSPCreate line 557 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c [0]PETSC ERROR: [0] PCMGSetLevels line 180 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c Gaetan On Fri, Aug 31, 2012 at 11:03 PM, Gaetan Kenway wrote: > Hi again > > I distilled the problem down further to a simple subroutine in mgtest.F, > copied directly from ex8f.F > > subroutine MGTest() > implicit none > > #include "include/finclude/petsc.h" > > KSP ksp > PC pc > PetscErrorCode ierr > PetscInt nlevels, two > two = 2 > > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCMG,ierr) > call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) > > call KSPDestroy(ksp) > > end subroutine MGTest > > When I ran this simple case with gdb I got the following backtrace: > > #0 0xb694d465 in PMPI_Attr_get () from /usr/local/lib/libmpi.so.0 > #1 0xb555d7c9 in PetscCommDuplicate (comm_in=0x0, comm_out=0x88d7378, > first_tag=0x88d73a0) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c:145 > #2 0xb5562ef7 in PetscHeaderCreate_Private (h=0x88d7370, classid=1211235, > type=-1, class_name=0xb61cfd70 "KSP", descr=0xb61cfd74 "Krylov Method", > mansec=0xb61cfd70 "KSP", comm=0x0, > des=0xb5eeac70 , vie=0xb5efafd7 ) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c:51 > #3 0xb5eff7ca in KSPCreate (comm=0x0, inksp=0x88d6ee0) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c:564 > #4 0xb5d4ac3a in PCMGSetLevels (pc=0x89089a0, levels=2, comms=0xb4bbf5a0) > at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c:219 > #5 0xb5d2c73c in pcmgsetlevels_ (pc=0xbfffec98, levels=0xbfffec8c, > comms=0xb4bbf5a0, ierr=0xbfffec94) at > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/ftn-custom/zmgf.c:15 > #6 0xb4affe1d in mgtest () at mgtest.F:15 > #7 0xb4aad769 in f2py_rout_warp_mgtest (capi_self=0x8618758, > capi_args=0xb7c3002c, capi_keywds=0x0, f2py_func=0xb4affdc0 ) at > warpmodule.c:4273 > #8 0xb4aa58cf in fortran_call (fp=0x8618758, arg=0xb7c3002c, kw=0x0) at > /usr/lib/python2.6/dist-packages/numpy/f2py/src/fortranobject.c:318 > #9 0x0805fd6a in PyObject_Call () > #10 0x080dd5b0 in PyEval_EvalFrameEx () > #11 0x080dfbb2 in PyEval_EvalCodeEx () > #12 0x080dfca7 in PyEval_EvalCode () > #13 0x080fd956 in PyRun_FileExFlags () > #14 0x080fdbb2 in PyRun_SimpleFileExFlags () > #15 0x0805b6d3 in Py_Main () > #16 0x0805a8ab in main () > > The only difference is that I'm calling the routine from python and petsc > is initialized from petsc4py. > > Any ideas? > > Thanks, > Gaetan > > > On Fri, Aug 31, 2012 at 9:48 PM, Jed Brown wrote: > >> src/ksp/pc/examples/tests/ex8f.F is a simple example, but I made the >> modification below and it runs fine. >> >> diff --git a/src/ksp/pc/examples/tests/ex8f.F >> b/src/ksp/pc/examples/tests/ex8f.F >> --- a/src/ksp/pc/examples/tests/ex8f.F >> +++ b/src/ksp/pc/examples/tests/ex8f.F >> @@ -35,7 +35,7 @@ >> Vec x,b,u >> PC pc >> PetscInt n,dim,istart,iend >> - PetscInt i,j,jj,ii,one,zero >> + PetscInt i,j,jj,ii,one,two,zero >> PetscErrorCode ierr >> PetscScalar v >> external MyResidual >> @@ -50,6 +50,7 @@ >> pfive = .5d0 >> n = 6 >> dim = n*n >> + two = 2 >> one = 1 >> zero = 0 >> >> @@ -138,7 +139,7 @@ >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> call KSPGetPC(ksp,pc,ierr) >> call PCSetType(pc,PCMG,ierr) >> - call PCMGSetLevels(pc,one,PETSC_NULL_OBJECT,ierr) >> + call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) >> call PCMGSetResidual(pc,zero,PCMGDefaultResidual,A,ierr) >> >> call PCMGSetResidual(pc,zero,MyResidual,A,ierr) >> @@ -147,6 +148,7 @@ >> ! also serves as the preconditioning matrix. >> >> call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr) >> + call KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD,ierr) >> >> >> call KSPDestroy(ksp,ierr) >> >> $ mpirun.hydra -n 2 ./ex8f >> KSP Object: 2 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using DEFAULT norm type for convergence test >> PC Object: 2 MPI processes >> type: mg >> MG: type is MULTIPLICATIVE, levels=2 cycles=v >> Cycles per PCApply=1 >> Not using Galerkin computed coarse grid matrices >> Coarse grid solver -- level ------------------------------- >> KSP Object: (mg_coarse_) 2 MPI processes >> type: preonly >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (mg_coarse_) 2 MPI processes >> type: redundant >> Redundant preconditioner: Not yet setup >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (mg_levels_1_) 2 MPI processes >> type: chebyshev >> Chebyshev: eigenvalue estimates: min = 0, max = 0 >> maximum iterations=2, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (mg_levels_1_) 2 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix = precond matrix: >> Matrix Object: 2 MPI processes >> type: mpiaij >> rows=36, cols=36 >> total: nonzeros=156, allocated nonzeros=252 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> >> >> >> On Fri, Aug 31, 2012 at 7:27 PM, Gaetan Kenway wrote: >> >>> Hi >>> >>> I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to >>> use PCMG to precondition a linear system of equations resulting from the >>> discretization of the EUler equations. It is from a multi-block code so I >>> have the ability to easily generate restriction operators using geometric >>> method. >>> >>> I'm currently trying to setup a simple 2V cycle. For this I use: >>> >>> ! Create the restriction operator between finest level and one below >>> call createRestrictOperator(RL1, 1) >>> >>> call PCSetFromOptions(pc, PETScIerr) >>> nlevels=2 >>> !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't >>> work from fortran!!! >>> call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr) >>> call PCMGSetRestriction(pc, 1, RL1, PETScierr) >>> >>> call PCView(pc, PETScierr) >>> >>> I run with the command line argument -pc_mg_levels 2 . >>> >>> First issue: PCMGSetLevels does not work correctly from Fortran. The >>> traceback I get when I run (with the commented-out line above and without >>> the command line arguments) is: >>> >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try >>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >>> corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscCommDuplicate line 144 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c >>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c >>> [0]PETSC ERROR: [0] KSPCreate line 557 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c >>> [0]PETSC ERROR: [0] PCMGSetLevels line 180 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c >>> >>> I can get around that issue by using -pc_mg_levels 2 and not calling the >>> PCMGSetLevels() command, but I would like to actually use this command and >>> not have set it via manually setting an option. The result of the PCView is >>> listed below which appears to be what I expect (given the default options) >>> >>> PC Object: 1 MPI processes >>> type: mg >>> MG: type is MULTIPLICATIVE, levels=2 cycles=v >>> Cycles per PCApply=1 >>> Using Galerkin computed coarse grid matrices >>> Coarse grid solver -- level ------------------------------- >>> KSP Object: (mg_coarse_) 1 MPI processes >>> type: preonly >>> maximum iterations=1, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: (mg_coarse_) 1 MPI processes >>> type: lu >>> LU: out-of-place factorization >>> tolerance for zero pivot 2.22045e-14 >>> matrix ordering: nd >>> Down solver (pre-smoother) on level 1 ------------------------------- >>> KSP Object: (mg_levels_1_) 1 MPI processes >>> type: chebyshev >>> Chebyshev: eigenvalue estimates: min = 0, max = 0 >>> maximum iterations=2, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: (mg_levels_1_) 1 MPI processes >>> type: sor >>> SOR: type = local_symmetric, iterations = 1, local iterations = >>> 1, omega = 1 >>> Up solver (post-smoother) same as down solver (pre-smoother) >>> linear system matrix followed by preconditioner matrix: >>> Matrix Object: 1 MPI processes >>> type: seqbaij >>> rows=92160, cols=92160, bs=5 >>> total: nonzeros=5582400, allocated nonzeros=5582400 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 5 >>> Matrix Object: 1 MPI processes >>> type: seqbaij >>> rows=92160, cols=92160, bs=5 >>> total: nonzeros=3089600, allocated nonzeros=3089600 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 5 >>> >>> >>> So that's ok. When it gets to actually solving my system with KSPSolve() >>> I get a returned error code 73 which is "object in argument is in wrong >>> state, e.g. unassembled mat" >>> >> >> In general, please don't paraphrase error messages. >> >> With these problems, try running in a debugger to get a real trace and >> with valgrind. The most common problem is stack corruption due to passing >> the wrong number or types of arguments. >> >> >>> >>> and the traceback is: >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] MatGetVecs line 8142 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: [0] KSPGetVecs line 774 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c >>> [0]PETSC ERROR: [0] PCSetUp_MG line 508 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c >>> [0]PETSC ERROR: [0] PCSetUp line 810 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: [0] KSPSetUp line 182 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: [0] KSPSolve line 351 >>> /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c >>> >>> Which would indicate to me its an issue with one of the matrices >>> involved, either the linear operator, A, the preconditioner P or my >>> provided restriction matrix RL1. A and P i'm pretty sure are OK, since it >>> works just fine when I use my usual BlockJacobi/Additive Schwartz with >>> ILU(k) on the subdomains. Which leaves the Restriction operator as the >>> issue. I added the following code before PCView() >>> >>> call MatGetVecs(RL1, vec1, vec2, ierr) >>> call VecSet(vec1, 1.0, ierr) >>> call MatMult(RL1, vec1, vec2, ierr) >>> >>> which ran through just fine (I check each of the ierrs but those are not >>> shown). >>> >>> So I'm at a loss as to what is causing the error. >>> I'm using an up-to-date copy of petsc-3.3 from the repository. >>> >>> Thanks, >>> >>> Gaetan >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Aug 31 22:06:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 31 Aug 2012 22:06:47 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 10:05 PM, Jed Brown wrote: > On Fri, Aug 31, 2012 at 10:03 PM, Gaetan Kenway wrote: > >> I distilled the problem down further to a simple subroutine in mgtest.F, >> copied directly from ex8f.F >> >> subroutine MGTest() >> implicit none >> >> #include "include/finclude/petsc.h" >> >> KSP ksp >> PC pc >> PetscErrorCode ierr >> PetscInt nlevels, two >> two = 2 >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> call KSPGetPC(ksp,pc,ierr) >> call PCSetType(pc,PCMG,ierr) >> call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr) >> >> call KSPDestroy(ksp) >> > > You are certainly missing an argument (ierr) to KSPDestroy... > True, but the backtrace also shows that comm = 0x0 on the call to KSPCreate(), which leads me to believe that your petsc4py has not initialized PETSc, and therefor not initialized PETSC_COMM_WORLD. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 31 22:08:45 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 22:08:45 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 10:06 PM, Matthew Knepley wrote: > True, but the backtrace also shows that comm = 0x0 on the call to > KSPCreate(), which > leads me to believe that your petsc4py has not initialized PETSc, and > therefor not > initialized PETSC_COMM_WORLD. > Could catch, maybe PetscFunctionBegin() should check that PETSc has been initialized. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpau at lbl.gov Fri Aug 31 23:04:44 2012 From: gpau at lbl.gov (George Pau) Date: Fri, 31 Aug 2012 21:04:44 -0700 Subject: [petsc-users] pctype hmpi In-Reply-To: <8883F05E-51DE-4B37-82BB-102A3925505E@mcs.anl.gov> References: <560D180A-A4E6-4D84-BFBB-FA3AF21535B1@mcs.anl.gov> <8A172C6B-2B46-49BC-BB96-DA069CF8A624@lbl.gov> <396C9322-5C95-485A-9235-786ABFC3F190@mcs.anl.gov> <2EABC8B5-DF3A-4B05-A49E-E26566C31979@mcs.anl.gov> <8883F05E-51DE-4B37-82BB-102A3925505E@mcs.anl.gov> Message-ID: Hi Barry, It works now. Thank you so much for the help. Thanks, George On Fri, Aug 31, 2012 at 7:57 PM, Barry Smith wrote: > > Ok, another bug. Put the attached file in src/mat/impls/aij/mpi and run > make in that directory then relink program and run again. > > Barry > > [see attached file: mpiaij.c] > > On Aug 31, 2012, at 5:26 PM, George Pau wrote: > > > hi Barry, > > > > The hmpi option is read in properly now. The error is now different > when I use -hmpi_spawn_size 3 with mpiexec -n 1. My printout suggests this > is now happening in the KSPSolve. > > > > George > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 1 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > [0] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [0] petscinitialize_(): Running on machine: gilbert > > [1] petscinitialize_(): (Fortran):PETSc successfully started: procs 2 > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > [1] petscinitialize_(): Running on machine: gilbert > > [0] PetscHMPISpawn(): PETSc HMPI successfully spawned: number of nodes = > 1 node size = 3 > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374781 max tags = 2147483647 > > [0] MatSetUp(): Warning not preallocating matrix storage > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage space: 3978 > unneeded,3222 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is > 360 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using Inode > routines > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374781 > > start ksp > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374781 > > [0] PCSetUp(): Setting up new PC > > [0] PetscCommDuplicate(): Duplicating a communicator -2080374782 > -2080374780 max tags = 2147483647 > > [0] PetscCommDuplicate(): Duplicating a communicator -1006632960 > -1006632959 max tags = 2147483647 > > [1] PetscCommDuplicate(): Duplicating a communicator -2080374779 > -2080374778 max tags = 2147483647 > > [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 > -2080374780 > > [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 > -1006632959 > > [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 > -2080374778 > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374779 max tags = 2147483647 > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -1006632958 max tags = 2147483647 > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374777 max tags = 2147483647 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374779 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -1006632958 > > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > > [0] Petsc_DelComm(): Removing reference to PETSc communicator imbedded > in a user MPI_Comm m -2080374779 > > [0] Petsc_DelComm(): User MPI_Comm m 1140850689 is being freed, removing > reference from inner PETSc comm to this outer comm > > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374779 > > [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374779 > > [0] PetscCommDuplicate(): Using internal PETSc communicator -2080374782 > -2080374780 > > [0] PetscCommDuplicate(): Using internal PETSc communicator -1006632960 > -1006632959 > > [1] PetscCommDuplicate(): Using internal PETSc communicator -2080374779 > -2080374778 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -1006632958 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374777 > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > Caught signal number 11 SEGV: Segmentation Violation, probably memory > access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > [1]PETSC ERROR: likely location of problem given in stack below > > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [1]PETSC ERROR: INSTEAD the line number of the start of the > function > > [1]PETSC ERROR: is given. > > [1]PETSC ERROR: [1] MatDistribute_MPIAIJ line 192 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: [1] PCSetUp_HMPI_MP line 90 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > [1]PETSC ERROR: [1] PetscHMPIHandle line 253 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > > [1]PETSC ERROR: [1] PetscHMPISpawn line 71 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [1]PETSC ERROR: Signal received! > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [1]PETSC ERROR: See docs/index.html for manual pages. > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 15:20:27 2012 > > [1]PETSC ERROR: [0]PETSC ERROR: Note: The EXACT line numbers in the > stack are not available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > > [0]PETSC ERROR: is given. > > Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [1]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 > > [1]PETSC ERROR: Configure options --with-debugging=1 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > > [0]PETSC ERROR: [0] MatDistribute_MPIAIJ line 192 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: [0] PCSetUp_HMPI_MP line 90 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > [0]PETSC ERROR: [0] PetscHMPIHandle line 253 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > > [0]PETSC ERROR: [0] PetscHMPISpawn line 71 > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/sys/objects/mpinit.c > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named gilbert > by gpau Fri Aug 31 15:20:27 2012 > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > [0]PETSC ERROR: Configure run at Fri Aug 31 15:16:04 2012 > > [0]PETSC ERROR: Configure options --with-debugging=1 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > Fatal error in MPI_Allreduce: Other MPI error, error stack: > > MPI_Allreduce(855)........: MPI_Allreduce(sbuf=0x7fff6315ead0, > rbuf=0x7fff6315eae0, count=2, MPI_INT, MPI_MAX, comm=0x84000004) failed > > MPIR_Allreduce_impl(712)..: > > MPIR_Allreduce_intra(534).: > > dequeue_and_set_error(596): Communication error with rank 0 > > > > > > On Fri, Aug 31, 2012 at 2:09 PM, Barry Smith wrote: > > [see attached file: zstart.c] > > > > On Aug 31, 2012, at 4:07 PM, George Pau wrote: > > > > > Hi Barry, > > > > > > You forgot the file ... > > > > > > George > > > > > > On Fri, Aug 31, 2012 at 2:04 PM, Barry Smith > wrote: > > > > > > Yikes. It is totally my fault. The handling of these merge and > spawn options is done only for PetscInitialize() for C. Not for Fortran, > hence the arguments just got ignored. > > > > > > Please find attached a file zstart.c put it in the directory > src/sys/ftn-custom and run make in that directory (with appropriate > PETSC_DIR and PETSC_ARCH set) > > > > > > Then link and run the example again. > > > > > > > > > Barry > > > > > > > > > On Aug 31, 2012, at 3:30 PM, George Pau wrote: > > > > > > > Sorry, it was a cut and paste error. I tried running the code with > all the options in the command line: > > > > > > > > mpiexec.mpich2 -n 1 xt2_eos4 -hmpi_spawn_size 3 -pc_type hmpi > -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre > boomeramg > > > > > > > > mpiexec.mpich2 -n 2 xt2_eos4 -hmpi_merge_size 2 -pc_type hmpi > -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre > boomeramg > > > > > > > > but I get the exact same outputs. > > > > > > > > George > > > > > > > > > > > > > > > > On Fri, Aug 31, 2012 at 1:18 PM, Barry Smith > wrote: > > > > > > > > On Aug 31, 2012, at 3:09 PM, George Pau wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > For the hmpi_spawn_size, the options in my .petscrc are > > > > > -info > > > > > -pc_view > > > > > pc_type hmpi > > > > > > > > How come there is no - in front of this one? > > > > > > > > > -ksp_type preonly > > > > > -ksp_view > > > > > -hmpi_pc_monitor > > > > > -hmpi_ksp_monitor > > > > > -hmpi_ksp_type cg > > > > > -hmpi_pc_type hypre > > > > > -hmpi_pc_hypre_type boomeramg > > > > > -hmpi_spawn_size 3 > > > > > > > > > > mpiexec.mpich2 -n 1 myprogram > > > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: > procs 1 > > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 360 X 360; storage > space: 3978 unneeded,3222 used > > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 360 > > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > > [0] Mat_CheckInode(): Found 120 nodes of 360. Limit used: 5. Using > Inode routines > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > > > > > > > > > > Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > > PMPI_Bcast(1478): MPI_Bcast(buf=0x7fff30dacecc, count=1, MPI_INT, > root=0, comm=0x0) failed > > > > > PMPI_Bcast(1418): Invalid communicator > > > > > > > > > > I inserted some print statement between the ksp calls and found > that the error occurs in > > > > > > > > > > call KSPSetFromOptions(ksp, pierr) > > > > > > > > > > 2. If I change hmpi_spawn_size 3 to hmpi_merge_size 2 and launch > my job by > > > > > > > > How come there is no - in front of hmpi_merge_size 2? > > > > > > > > > > > > Can you try putting all the arguments as command line arguments > instead of in a file? It shouldn't matter but it seems like some of the > arguments are being ignored. > > > > > > > > Barry > > > > > > > > > > > > > > > > > > mpiexec.mpich2 -n 2 myprogram > > > > > > > > > > [0] petscinitialize_(): (Fortran):PETSc successfully started: > procs 2 > > > > > [0] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > > [0] petscinitialize_(): Running on machine: gilbert > > > > > [1] petscinitialize_(): (Fortran):PETSc successfully started: > procs 2 > > > > > [1] PetscGetHostName(): Rejecting domainname, likely is NIS > gilbert.(none) > > > > > [1] petscinitialize_(): Running on machine: gilbert > > > > > > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374780 max tags = 2147483647 > > > > > [0] MatSetUp(): Warning not preallocating matrix storage > > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374782 max tags = 2147483647 > > > > > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374777 max tags = 2147483647 > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374780 max tags = 2147483647 > > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > > [0] MatStashScatterBegin_Private(): No of messages: 1 > > > > > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 12896 > > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 1611 entries, uses 0 > mallocs. > > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. > > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage > space: 1998 unneeded,1602 used > > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 180 > > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > > [0] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 180; storage > space: 1998 unneeded,1602 used > > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 180 > > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 9 > > > > > [1] Mat_CheckInode(): Found 60 nodes of 180. Limit used: 5. Using > Inode routines > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374777 > > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374780 > > > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > > > > [0] VecScatterCreate(): General case: MPI to Seq > > > > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: > 396 unneeded,9 used > > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 3 > > > > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374782 > > > > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 180 X 3; storage space: > 396 unneeded,9 used > > > > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 3 > > > > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3 > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > > [0] VecAssemblyBegin_MPI(): Stash has 180 entries, uses 1 mallocs. > > > > > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 > mallocs. > > > > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374782 > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > > [0] PCSetUp(): Setting up new PC > > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374780 > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > > > [0]PETSC ERROR: Nonconforming object sizes! > > > > > [0]PETSC ERROR: HMPI preconditioner only works for sequential > solves! > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named > gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: PCCreate_HMPI() line 283 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/impls/openmp/hpc.c > > > > > [0]PETSC ERROR: PCSetType() line 83 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > > [0]PETSC ERROR: PCSetFromOptions() line 188 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/pcset.c > > > > > [0]PETSC ERROR: KSPSetFromOptions() line 287 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itcl.c > > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > > > > [0]PETSC ERROR: No support for this operation for this object type! > > > > > [0]PETSC ERROR: PC does not have apply! > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 > 11:26:24 CDT 2012 > > > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: ../../esd-tough2/xt2_eos4 on a arch-linu named > gilbert by gpau Fri Aug 31 13:00:31 2012 > > > > > [0]PETSC ERROR: Libraries linked from > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/lib > > > > > [0]PETSC ERROR: Configure run at Thu Aug 30 15:27:17 2012 > > > > > [0]PETSC ERROR: Configure options --with-debugging=0 > --with-mpi-dir=/usr/lib/mpich2 --download-hypre=1 > --prefix=/home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > > [0]PETSC ERROR: PCApply() line 382 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/pc/interface/precon.c > > > > > [0]PETSC ERROR: KSPInitialResidual() line 64 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itres.c > > > > > [0]PETSC ERROR: KSPSolve_GMRES() line 230 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/impls/gmres/gmres.c > > > > > [0]PETSC ERROR: KSPSolve() line 446 in > /home/gpau/tough_codes/esd-tough2/build/Linux-x86_64-Debug-MPI-EOS4/toughlib/tpls/petsc/petsc-3.3-p3-source/src/ksp/ksp/interface/itfunc.c > > > > > > > > > > I note that the error appears to occur at the same point. > > > > > > > > > > George > > > > > > > > > > > > > > > On Fri, Aug 31, 2012 at 11:31 AM, Barry Smith > wrote: > > > > > > > > > > On Aug 31, 2012, at 1:27 PM, George Pau wrote: > > > > > > > > > > > Hi Barry, > > > > > > > > > > > > 1. It is the exact same error related to MPI_ERR_COMM and > MPI_Bcast. > > > > > > > > > > That should not happen. Please run and send all the output > including the exact command line used > > > > > > > > > > > > > > > > I am currently using the MPICH2 distribution provided by ubuntu > but if MPICH version that Petsc download with -download-mpich works, I can > use that. > > > > > > 2. If I use hmpi_merge_size, I will need to launch mpiexec with > more than 1 cpus. But, petsc will complain that the pctype hmpi can only > be used in serial. > > > > > > > > > > That should not happen. Run with 2 MPI processes and > -hmpi_merge_size 2 and send the complete error message. > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > > > > On Aug 31, 2012, at 11:17 AM, Barry Smith wrote: > > > > > > > > > > > >> > > > > > >> On Aug 30, 2012, at 10:02 PM, George Pau wrote: > > > > > >> > > > > > >>> Hi Barry, > > > > > >>> > > > > > >>> I tried with the addition of > > > > > >>> > > > > > >>> -hmpi_spawn_size 3 > > > > > >>> > > > > > >>> but I am still getting the same error though. > > > > > >> > > > > > >> The EXACT same error? Or some other error? > > > > > >> > > > > > >> What happens if you run with the -hmpi_merge_size > option instead? > > > > > >> > > > > > >> Barry > > > > > >> > > > > > >> 1) I am getting a crash with the spawn version I suspect is due > to bugs in the MPICH version I am using related to spawn. > > > > > >> > > > > > >> 2) I am getting errors with the merge version due to Apple's > ASLR which they make hard to turn off. > > > > > >> > > > > > >> > > > > > >>> I am using mpich2. Any other options to try? > > > > > >>> > > > > > >>> George > > > > > >>> > > > > > >>> > > > > > >>> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote: > > > > > >>> > > > > > >>>> > > > > > >>>> On Aug 30, 2012, at 7:24 PM, George Pau wrote: > > > > > >>>> > > > > > >>>>> Hi, > > > > > >>>>> > > > > > >>>>> I have some issues using the -pctype hmpi. I used the same > setting found at > > > > > >>>>> > > > > > >>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html > > > > > >>>>> > > > > > >>>>> i.e. > > > > > >>>>> -pc_type hmpi > > > > > >>>>> -ksp_type preonly > > > > > >>>>> -hmpi_ksp_type cg > > > > > >>>>> -hmpi_pc_type hypre > > > > > >>>>> -hmpi_pc_hypre_type boomeramg > > > > > >>>>> > > > > > >>>>> My command is > > > > > >>>>> > > > > > >>>>> mpiexec -n 1 myprogram > > > > > >>>> > > > > > >>>> Sorry the documentation doesn't make this clearer. You need > to start PETSc with special options to get the "worker" processes > initialized. From the manual page for PCHMPI it has > > > > > >>>> > > > > > >>>> See PetscHMPIMerge() and PetscHMPISpawn() for two ways to > start up MPI for use with this preconditioner > > > > > >>>> > > > > > >>>> This will tell you want option to start PETSc up with. > > > > > >>>> > > > > > >>>> I will fix the PC so that it prints a far more useful error > message. > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> Barry > > > > > >>>> > > > > > >>>> > > > > > >>>>> > > > > > >>>>> But, I get > > > > > >>>>> > > > > > >>>>> [gilbert:4041] *** An error occurred in MPI_Bcast > > > > > >>>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD > > > > > >>>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator > > > > > >>>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will > now abort) > > > > > >>>>> > > > > > >>>>> with openmpi. I get similar error with mpich2 > > > > > >>>>> > > > > > >>>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack: > > > > > >>>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, > MPI_INT, root=0, comm=0x0) failed > > > > > >>>>> PMPI_Bcast(1418): Invalid communicator > > > > > >>>>> > > > > > >>>>> I couldn't figure out what is wrong. My petsc is version > 3.3.3 and the configuration is -with-debugging=0 > --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu > machine. > > > > > >>>>> > > > > > >>>>> Note that with the default pc_type and ksp_type, everything > is fine. It was also tested with multiple processors. I wondering whether > there are some options that I am not specifying correctly? > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> George Pau > > > > > >>>>> Earth Sciences Division > > > > > >>>>> Lawrence Berkeley National Laboratory > > > > > >>>>> One Cyclotron, MS 74-120 > > > > > >>>>> Berkeley, CA 94720 > > > > > >>>>> > > > > > >>>>> (510) 486-7196 > > > > > >>>>> gpau at lbl.gov > > > > > >>>>> http://esd.lbl.gov/about/staff/georgepau/ > > > > > >>>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > George Pau > > > > > Earth Sciences Division > > > > > Lawrence Berkeley National Laboratory > > > > > One Cyclotron, MS 74-120 > > > > > Berkeley, CA 94720 > > > > > > > > > > (510) 486-7196 > > > > > gpau at lbl.gov > > > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > George Pau > > > > Earth Sciences Division > > > > Lawrence Berkeley National Laboratory > > > > One Cyclotron, MS 74-120 > > > > Berkeley, CA 94720 > > > > > > > > (510) 486-7196 > > > > gpau at lbl.gov > > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > > > > > > > > > -- > > > George Pau > > > Earth Sciences Division > > > Lawrence Berkeley National Laboratory > > > One Cyclotron, MS 74-120 > > > Berkeley, CA 94720 > > > > > > (510) 486-7196 > > > gpau at lbl.gov > > > http://esd.lbl.gov/about/staff/georgepau/ > > > > > > > > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Fri Aug 31 23:22:44 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Sat, 1 Sep 2012 00:22:44 -0400 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: Hi Again I also tried petsc-3.2 version and I still get the same backtrace. If its not possible to figure out where the communicator segfault is coming from its not a huge deal...I've just set the option using PetscOptionsSetValue() and then use PCSetFromOptions() to pull it back out. That seems to work fine. Even avoiding the above problem, with the PetscOptionsSetValue I'm still receiving an error code 73 when I run the multigrid solver. I've included the backtrace output below but its not a lot of help since the code exited cleaning using my error checking procedure ================================================================= PETSc or MPI Error. Error Code 73. Detected on Proc 0 Error at line: 122 in file: solveADjointTransposePETSc.F90 ================================================================= Program received signal SIGSEGV, Segmentation fault. 0xb68c98f0 in PMPI_Abort () from /usr/local/lib/libmpi.so.0 (gdb) bt #0 0xb68c98f0 in PMPI_Abort () from /usr/local/lib/libmpi.so.0 #1 0xb519f190 in pmpi_abort__ () from /usr/local/lib/libmpi_f77.so.0 #2 0xb44c9e4c in echk (ierr=@0x49, file=..., line=@0x7a, .tmp.FILE.len_V$eb=30) at terminate.f90:154 #3 0xb44bd68f in solveadjointtransposepetsc () at solveADjointTransposePETSc.F90:122 #4 0xb44138a9 in f2py_rout_sumb_solveadjointtransposepetsc () from /tmp/tmpKYF_DT/sumb.so #5 0xb440fd35 in fortran_call () from /tmp/tmpKYF_DT/sumb.so #6 0x0805fd6a in PyObject_Call () #7 0x080dd5b0 in PyEval_EvalFrameEx () #8 0x080dfbb2 in PyEval_EvalCodeEx () #9 0x08168f1f in ?? () #10 0x0805fd6a in PyObject_Call () #11 0x080dcbeb in PyEval_EvalFrameEx () #12 0x080dfbb2 in PyEval_EvalCodeEx () #13 0x080de145 in PyEval_EvalFrameEx () #14 0x080dfbb2 in PyEval_EvalCodeEx () #15 0x080dfca7 in PyEval_EvalCode () #16 0x080fd956 in PyRun_FileExFlags () #17 0x080fdbb2 in PyRun_SimpleFileExFlags () #18 0x0805b6d3 in Py_Main () #19 0x0805a8ab in main () Valgrid was clean right up until the end where I get the normal error message: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatGetVecs line 8142 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: [0] KSPGetVecs line 774 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: [0] PCSetUp_MG line 508 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCSetUp line 810 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 182 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPSolve line 351 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c I will try it on a different system tomorrow to see if I have any more luck. Thanks, Gaetan On Fri, Aug 31, 2012 at 11:08 PM, Jed Brown wrote: > On Fri, Aug 31, 2012 at 10:06 PM, Matthew Knepley wrote: > >> True, but the backtrace also shows that comm = 0x0 on the call to >> KSPCreate(), which >> leads me to believe that your petsc4py has not initialized PETSc, and >> therefor not >> initialized PETSC_COMM_WORLD. >> > > Could catch, maybe PetscFunctionBegin() should check that PETSc has been > initialized. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 31 23:29:45 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 31 Aug 2012 23:29:45 -0500 Subject: [petsc-users] MG Preconditioning In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 11:22 PM, Gaetan Kenway wrote: > Hi Again > > I also tried petsc-3.2 version and I still get the same backtrace. > > If its not possible to figure out where the communicator segfault is > coming from its not a huge deal...I've just set the option using > PetscOptionsSetValue() and then use PCSetFromOptions() to pull it back out. > That seems to work fine. > > Even avoiding the above problem, with the PetscOptionsSetValue I'm still > receiving an error code 73 when I run the multigrid solver. I've included > the backtrace output below but its not a lot of help since the code exited > cleaning using my error checking procedure > > ================================================================= > PETSc or MPI Error. Error Code 73. Detected on Proc 0 > Error at line: 122 in file: solveADjointTransposePETSc.F90 > ================================================================= > > Program received signal SIGSEGV, Segmentation fault. > 0xb68c98f0 in PMPI_Abort () from /usr/local/lib/libmpi.so.0 > (gdb) bt > #0 0xb68c98f0 in PMPI_Abort () from /usr/local/lib/libmpi.so.0 > #1 0xb519f190 in pmpi_abort__ () from /usr/local/lib/libmpi_f77.so.0 > #2 0xb44c9e4c in echk (ierr=@0x49, file=..., line=@0x7a, > .tmp.FILE.len_V$eb=30) at terminate.f90:154 > #3 0xb44bd68f in solveadjointtransposepetsc () at > solveADjointTransposePETSc.F90:122 > #4 0xb44138a9 in f2py_rout_sumb_solveadjointtransposepetsc () from > /tmp/tmpKYF_DT/sumb.so > #5 0xb440fd35 in fortran_call () from /tmp/tmpKYF_DT/sumb.so > #6 0x0805fd6a in PyObject_Call () > #7 0x080dd5b0 in PyEval_EvalFrameEx () > #8 0x080dfbb2 in PyEval_EvalCodeEx () > #9 0x08168f1f in ?? () > #10 0x0805fd6a in PyObject_Call () > #11 0x080dcbeb in PyEval_EvalFrameEx () > #12 0x080dfbb2 in PyEval_EvalCodeEx () > #13 0x080de145 in PyEval_EvalFrameEx () > #14 0x080dfbb2 in PyEval_EvalCodeEx () > #15 0x080dfca7 in PyEval_EvalCode () > #16 0x080fd956 in PyRun_FileExFlags () > #17 0x080fdbb2 in PyRun_SimpleFileExFlags () > #18 0x0805b6d3 in Py_Main () > #19 0x0805a8ab in main () > This stack doesn't involve PETSc at all. > > Valgrid was clean right up until the end where I get the normal error > message: > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatGetVecs line 8142 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] KSPGetVecs line 774 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: [0] PCSetUp_MG line 508 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCSetUp line 810 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 182 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 351 > /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c > > I will try it on a different system tomorrow to see if I have any more > luck. > > Thanks, > > Gaetan > > > > On Fri, Aug 31, 2012 at 11:08 PM, Jed Brown wrote: > >> On Fri, Aug 31, 2012 at 10:06 PM, Matthew Knepley wrote: >> >>> True, but the backtrace also shows that comm = 0x0 on the call to >>> KSPCreate(), which >>> leads me to believe that your petsc4py has not initialized PETSc, and >>> therefor not >>> initialized PETSC_COMM_WORLD. >>> >> >> Could catch, maybe PetscFunctionBegin() should check that PETSc has been >> initialized. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: