From li76pan at yahoo.com Mon Jul 2 08:37:53 2007 From: li76pan at yahoo.com (li pan) Date: Mon, 2 Jul 2007 06:37:53 -0700 (PDT) Subject: memory allocation Message-ID: <660776.69201.qm@web36801.mail.mud.yahoo.com> Dear developers, last week I asked a question about memory allocation. I'm trying to comprehend it from another aspect. The question is, if I have a mesh (HEX8) of size 128 x 128 x 40. There are three components for each node. In case of displacement, it's (u,v,w). So there must be 128 x 128 x 40 x 3 = 1966080 degrees of freedom. In sparse pattern, each element must have 7 x 3 = 21 non-zeros, because each node has 6 neighbours, plus itself, it's 7 nodes, then multiply 3 components, we get 21. If everything is allright until now, we have 1966080 x 21 non-zeros in the system matrix. This corresponds 42 MB. I wonder why there was so much memory required by petsc, more than 2 GB. Probably, I'm wrong on some points. Please remind me. kind regards pan ____________________________________________________________________________________ Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. http://autos.yahoo.com/green_center/ From aja2111 at columbia.edu Mon Jul 2 13:58:22 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Mon, 2 Jul 2007 14:58:22 -0400 Subject: memory allocation In-Reply-To: <660776.69201.qm@web36801.mail.mud.yahoo.com> References: <660776.69201.qm@web36801.mail.mud.yahoo.com> Message-ID: <37604ab40707021158y4794e262y2a7b38a4abc3ad9c@mail.gmail.com> Dear Pan, I don't see everything going on here, but you have to account for around 1966080*(4 or 8) bytes + 1966080*21*(4 or 8) bytes of indexing information for storing the locations of the data in the Matrix, then if you're using doubles, 1966080*21*8 bytes of data information. Adding these up, I get something closer to 500 MB. If PETSc is consuming more than 2 GB, there's probably something else going on to consume all that memory. Are you using GMRES to solve the system? Try calling as simple a version of your code as you can that sets up the mesh and pass in the -log flag on the command line and report back with the results. ~A On 7/2/07, li pan wrote: > Dear developers, > last week I asked a question about memory allocation. > I'm trying to comprehend it from another aspect. The > question is, if I have a mesh (HEX8) of size 128 x 128 > x 40. There are three components for each node. In > case of displacement, it's (u,v,w). So there must be > 128 x 128 x 40 x 3 = 1966080 degrees of freedom. In > sparse pattern, each element must have 7 x 3 = 21 > non-zeros, because each node has 6 neighbours, plus > itself, it's 7 nodes, then multiply 3 components, we > get 21. If everything is allright until now, we have > 1966080 x 21 non-zeros in the system matrix. This > corresponds 42 MB. I wonder why there was so much > memory required by petsc, more than 2 GB. > Probably, I'm wrong on some points. Please remind me. > > kind regards > > pan > > > > > ____________________________________________________________________________________ > Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. > http://autos.yahoo.com/green_center/ > > From li76pan at yahoo.com Tue Jul 3 02:11:17 2007 From: li76pan at yahoo.com (li pan) Date: Tue, 3 Jul 2007 00:11:17 -0700 (PDT) Subject: memory allocation In-Reply-To: <37604ab40707021158y4794e262y2a7b38a4abc3ad9c@mail.gmail.com> Message-ID: <623603.76102.qm@web36805.mail.mud.yahoo.com> Dear Aron, thanx for your answer. I've found the reason. As you said, petsc need indexing information as well as the dada. Furthermore, each node has 27 neighbours, since I'm doing FEM calculation. I mistakened last time. see u pan --- Aron Ahmadia wrote: > Dear Pan, > > I don't see everything going on here, but you have > to account for > around 1966080*(4 or 8) bytes + 1966080*21*(4 or 8) > bytes of indexing > information for storing the locations of the data in > the Matrix, then > if you're using doubles, 1966080*21*8 bytes of data > information. > Adding these up, I get something closer to 500 MB. > If PETSc is > consuming more than 2 GB, there's probably something > else going on to > consume all that memory. Are you using GMRES to > solve the system? > > Try calling as simple a version of your code as you > can that sets up > the mesh and pass in the -log flag on the command > line and report back > with the results. > > ~A > > > On 7/2/07, li pan wrote: > > Dear developers, > > last week I asked a question about memory > allocation. > > I'm trying to comprehend it from another aspect. > The > > question is, if I have a mesh (HEX8) of size 128 x > 128 > > x 40. There are three components for each node. In > > case of displacement, it's (u,v,w). So there must > be > > 128 x 128 x 40 x 3 = 1966080 degrees of freedom. > In > > sparse pattern, each element must have 7 x 3 = 21 > > non-zeros, because each node has 6 neighbours, > plus > > itself, it's 7 nodes, then multiply 3 components, > we > > get 21. If everything is allright until now, we > have > > 1966080 x 21 non-zeros in the system matrix. This > > corresponds 42 MB. I wonder why there was so much > > memory required by petsc, more than 2 GB. > > Probably, I'm wrong on some points. Please remind > me. > > > > kind regards > > > > pan > > > > > > > > > > > ____________________________________________________________________________________ > > Park yourself in front of a world of choices in > alternative vehicles. Visit the Yahoo! Auto Green > Center. > > http://autos.yahoo.com/green_center/ > > > > > > ____________________________________________________________________________________ Moody friends. Drama queens. Your life? Nope! - their life, your story. Play Sims Stories at Yahoo! Games. http://sims.yahoo.com/ From tim at cevis.uni-bremen.de Tue Jul 3 09:16:17 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Tue, 3 Jul 2007 16:16:17 +0200 (CEST) Subject: Number of iterations for ASM and BJACOBI preconditioners In-Reply-To: <623603.76102.qm@web36805.mail.mud.yahoo.com> References: <623603.76102.qm@web36805.mail.mud.yahoo.com> Message-ID: Dear all, For the ASM and BJACOBI preconditioners, I wonder how I can adjust the number of iterations to be used. I don't find an appropriate option (nor a tolerance option). Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From knepley at gmail.com Tue Jul 3 09:53:46 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 3 Jul 2007 09:53:46 -0500 Subject: Number of iterations for ASM and BJACOBI preconditioners In-Reply-To: References: <623603.76102.qm@web36805.mail.mud.yahoo.com> Message-ID: On 7/3/07, Tim Kr?ger wrote: > Dear all, > > For the ASM and BJACOBI preconditioners, I wonder how I can adjust the > number of iterations to be used. I don't find an appropriate option > (nor a tolerance option). For the default setup, you would use -sub_ksp_max_it since both PCs set the options prefix of their local solvers to "sub_". Matt > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From billy at dem.uminho.pt Wed Jul 4 14:18:18 2007 From: billy at dem.uminho.pt (billy at dem.uminho.pt) Date: Wed, 4 Jul 2007 20:18:18 +0100 Subject: Interlaced vectors. Message-ID: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> Hi, I have a velocity vector (u,v,w) with 3 updates. Is there an efficient way to interlace the 3 vectors into 1 update without creating a new data structure? Billy. From dalcinl at gmail.com Wed Jul 4 16:12:38 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 4 Jul 2007 18:12:38 -0300 Subject: Interlaced vectors. In-Reply-To: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> References: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> Message-ID: On 7/4/07, billy at dem.uminho.pt wrote: > Hi, > > I have a velocity vector (u,v,w) with 3 updates. Is there an efficient way to > interlace the 3 vectors into 1 update without creating a new data structure? Look at VecStrideScatter, but not sure exactly what you need. Can elaborate a bit more? What are x,u,v? Sepparate PETSc Vec's, or just a Vec with consecutive entries? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From billy at dem.uminho.pt Thu Jul 5 03:19:10 2007 From: billy at dem.uminho.pt (billy at dem.uminho.pt) Date: Thu, 5 Jul 2007 09:19:10 +0100 Subject: Interlaced vectors. In-Reply-To: References: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> Message-ID: <1183623550.468ca97e46a2a@serv-g1.ccom.uminho.pt> Right now, they are 3 separate VecCreateGhost that are updated at the same time using 3 calls to VecGhostUpdateBegin and VecGhostUpdateEnd. I read that putting them into a single call might accelerate communication. The question what is the best way to do this? Billy. Citando Lisandro Dalcin : > On 7/4/07, billy at dem.uminho.pt wrote: > > Hi, > > > > I have a velocity vector (u,v,w) with 3 updates. Is there an efficient way > to > > interlace the 3 vectors into 1 update without creating a new data > structure? > > Look at VecStrideScatter, but not sure exactly what you need. Can > elaborate a bit more? What are x,u,v? Sepparate PETSc Vec's, or just a > Vec with consecutive entries? > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > From bsmith at mcs.anl.gov Thu Jul 5 16:39:12 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 5 Jul 2007 16:39:12 -0500 (CDT) Subject: Interlaced vectors. In-Reply-To: <1183623550.468ca97e46a2a@serv-g1.ccom.uminho.pt> References: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> <1183623550.468ca97e46a2a@serv-g1.ccom.uminho.pt> Message-ID: You can simply create the vector with VecCreateGhost() with n and N 3 times as big, nghost 3 times as big and the ghosts[] array three times as long. (filled with 3*ghosts[0], 3*ghosts[0]+1,3*ghosts[0]+2,3*ghosts[1],3*ghosts[1]+1,3*ghosts[1]+2, etc where ghosts is what you use currently. Barry On Thu, 5 Jul 2007, billy at dem.uminho.pt wrote: > > Right now, they are 3 separate VecCreateGhost that are updated at the same time > using 3 calls to VecGhostUpdateBegin and VecGhostUpdateEnd. I read that putting > them into a single call might accelerate communication. The question what is > the best way to do this? > > Billy. > > > Citando Lisandro Dalcin : > > > On 7/4/07, billy at dem.uminho.pt wrote: > > > Hi, > > > > > > I have a velocity vector (u,v,w) with 3 updates. Is there an efficient way > > to > > > interlace the 3 vectors into 1 update without creating a new data > > structure? > > > > Look at VecStrideScatter, but not sure exactly what you need. Can > > elaborate a bit more? What are x,u,v? Sepparate PETSc Vec's, or just a > > Vec with consecutive entries? > > > > -- > > Lisandro Dalc?n > > --------------- > > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > > Tel/Fax: +54-(0)342-451.1594 > > > > > > > From li76pan at yahoo.com Fri Jul 6 07:35:12 2007 From: li76pan at yahoo.com (li pan) Date: Fri, 6 Jul 2007 05:35:12 -0700 (PDT) Subject: memory problem Message-ID: <681402.27467.qm@web36804.mail.mud.yahoo.com> Dear developers, I still have problem with memory allocation. As example, I used mesh size of 128 x 128 x 35, and got the following error message, [0]PETSC ERROR: PetscMallocAlign() line 62 in src/sys/src/memory/mal.c [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0] Maximum memory PetscMalloc()ed 1815995752 maximum size of entire process 0 [0]PETSC ERROR: Memory requested 1082172680! How should I understand "Maximum memory "? Is it maximum memory size a process can allocate? kind regards pan ____________________________________________________________________________________ Bored stiff? Loosen up... Download and play hundreds of games for free on Yahoo! Games. http://games.yahoo.com/games/front From billy at dem.uminho.pt Fri Jul 6 07:36:00 2007 From: billy at dem.uminho.pt (billy at dem.uminho.pt) Date: Fri, 6 Jul 2007 13:36:00 +0100 Subject: Interlaced vectors. In-Reply-To: References: <1183576698.468bf27adab6d@serv-g1.ccom.uminho.pt> <1183623550.468ca97e46a2a@serv-g1.ccom.uminho.pt> Message-ID: <1183725360.468e37304f14e@serv-g1.ccom.uminho.pt> Thanks, I will try that. Billy. Quoting Barry Smith : > > You can simply create the vector with VecCreateGhost() with > n and N 3 times as big, nghost 3 times as big and the > ghosts[] array three times as long. (filled with 3*ghosts[0], > 3*ghosts[0]+1,3*ghosts[0]+2,3*ghosts[1],3*ghosts[1]+1,3*ghosts[1]+2, etc > where ghosts is what you use currently. > > Barry > > On Thu, 5 Jul 2007, billy at dem.uminho.pt wrote: > > > > > Right now, they are 3 separate VecCreateGhost that are updated at the same > time > > using 3 calls to VecGhostUpdateBegin and VecGhostUpdateEnd. I read that > putting > > them into a single call might accelerate communication. The question what > is > > the best way to do this? > > > > Billy. > > > > > > Citando Lisandro Dalcin : > > > > > On 7/4/07, billy at dem.uminho.pt wrote: > > > > Hi, > > > > > > > > I have a velocity vector (u,v,w) with 3 updates. Is there an efficient > way > > > to > > > > interlace the 3 vectors into 1 update without creating a new data > > > structure? > > > > > > Look at VecStrideScatter, but not sure exactly what you need. Can > > > elaborate a bit more? What are x,u,v? Sepparate PETSc Vec's, or just a > > > Vec with consecutive entries? > > > > > > -- > > > Lisandro Dalc?n > > > --------------- > > > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > > > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > > > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > > > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > > > Tel/Fax: +54-(0)342-451.1594 > > > > > > > > > > > > > From knepley at gmail.com Fri Jul 6 08:26:43 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Jul 2007 08:26:43 -0500 Subject: memory problem In-Reply-To: <681402.27467.qm@web36804.mail.mud.yahoo.com> References: <681402.27467.qm@web36804.mail.mud.yahoo.com> Message-ID: Here "maximum memory" means all the memory that has been PetscMalloced. Some of this might have been freed, but we do not keep track. However, it looks like you are trying to allocate 1G. Also, it looks like you truncated the stack trace. Can you send the entire trace to petsc-maint at mcs.anl.gov? Thanks, Matt On 7/6/07, li pan wrote: > Dear developers, > I still have problem with memory allocation. As > example, I used mesh size of 128 x 128 x 35, and got > the following error message, > [0]PETSC ERROR: PetscMallocAlign() line 62 in > src/sys/src/memory/mal.c > [0]PETSC ERROR: Out of memory. This could be due to > allocating > [0]PETSC ERROR: too large an object or bleeding by not > properly > [0]PETSC ERROR: destroying unneeded objects. > [0] Maximum memory PetscMalloc()ed 1815995752 maximum > size of entire process 0 > [0]PETSC ERROR: Memory requested 1082172680! > > How should I understand "Maximum memory "? Is it > maximum memory size a process can allocate? > > kind regards > > pan > > > > > ____________________________________________________________________________________ > Bored stiff? Loosen up... > Download and play hundreds of games for free on Yahoo! Games. > http://games.yahoo.com/games/front > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From jagruti.trivedi at navy.mil Mon Jul 9 16:54:14 2007 From: jagruti.trivedi at navy.mil (Trivedi, Jagruti CIV 470000D, 474300D) Date: Mon, 9 Jul 2007 14:54:14 -0700 Subject: FW: Problem building Petsc-2.3.2-p10 Message-ID: <7F49B2C8312BE84FBC6BE7349A7EC58F05433F2D@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> Hello There, I am building Petsc-2.3.2-p10 on Solaris 10 for i386 that I got from Solaris freeware. I have f2cblaslapack package downloaded and install and also mpich 1.2.7 installed. I have python from ActivePython 2.5.1.1 . I can not get Petsc build. I have gcc and g++ compiler that comes with sunfreeware gcc 3.3.2. I do not have F77 or f90 compiler. I give ./config/configure.py but make fails for f2cblas.a and f2clapack.a library. Thanks, ...Jagruti From knepley at gmail.com Mon Jul 9 17:06:13 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Jul 2007 17:06:13 -0500 Subject: FW: Problem building Petsc-2.3.2-p10 In-Reply-To: <7F49B2C8312BE84FBC6BE7349A7EC58F05433F2D@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> References: <7F49B2C8312BE84FBC6BE7349A7EC58F05433F2D@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> Message-ID: We cannot see what the problem is without configure.log. Also, you should be sending this to petsc-maint at mcs.anl.gov. Matt On 7/9/07, Trivedi, Jagruti CIV 470000D, 474300D wrote: > > > > Hello There, > > I am building Petsc-2.3.2-p10 on Solaris 10 for i386 that I got from > Solaris freeware. > > I have f2cblaslapack package downloaded and install and also mpich 1.2.7 > installed. > I have python from ActivePython 2.5.1.1 . > I can not get Petsc build. > > I have gcc and g++ compiler that comes with sunfreeware gcc 3.3.2. I do > not have > F77 or f90 compiler. > > I give > ./config/configure.py include> but make fails for f2cblas.a and f2clapack.a library. > > Thanks, > > ...Jagruti > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Mon Jul 9 17:06:26 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 9 Jul 2007 17:06:26 -0500 (CDT) Subject: FW: Problem building Petsc-2.3.2-p10 In-Reply-To: <7F49B2C8312BE84FBC6BE7349A7EC58F05433F2D@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> References: <7F49B2C8312BE84FBC6BE7349A7EC58F05433F2D@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> Message-ID: Please send configure.log to petsc-maint at mcs.anl.gov (not here). Barry On Mon, 9 Jul 2007, Trivedi, Jagruti CIV 470000D, 474300D wrote: > > > > Hello There, > > I am building Petsc-2.3.2-p10 on Solaris 10 for i386 that I got from > Solaris freeware. > > I have f2cblaslapack package downloaded and install and also mpich 1.2.7 > installed. > I have python from ActivePython 2.5.1.1 . > I can not get Petsc build. > > I have gcc and g++ compiler that comes with sunfreeware gcc 3.3.2. I do > not have > F77 or f90 compiler. > > I give > ./config/configure.py include> but make fails for f2cblas.a and f2clapack.a library. > > Thanks, > > ...Jagruti > > From bbiskebo at us.ibm.com Tue Jul 10 11:41:15 2007 From: bbiskebo at us.ibm.com (Brian Biskeborn) Date: Tue, 10 Jul 2007 09:41:15 -0700 Subject: Petsc on Blue Gene Message-ID: Hello, Does anyone here have experience with using Petsc on a Blue Gene system? I'm at IBM's Almaden Research Center and am currently involved in porting some code that uses Petsc to Blue Gene. I've managed to compile and run the code, and it seems to produce the correct results, but it generates many floating point alignment exceptions at runtime. I traced some of these warnings to a call to MatAssemblyBegin, so I'm thinking maybe the problem is in Petsc. The Blue Gene architecture requires 8- or 16-byte memory alignment for floating point operations, depending on whether they are issued in parallel, so I tried bumping up the alignment value in src/sys/src/memory/mal.c. This had no effect on the problem. I didn't write any of the code I'm porting, and I'm not too familiar with it yet, so I suppose the problem could really be anywhere. My question here is simply this: has anyone seen this problem? Does it sound like an issue with Petsc, or is it more likely to be in the top-level code? Thanks, and best regards, Brian From mbostandoust at yahoo.com Tue Jul 10 17:28:36 2007 From: mbostandoust at yahoo.com (Mehdi Bostandoost) Date: Tue, 10 Jul 2007 15:28:36 -0700 (PDT) Subject: non collective mpi_allreduce Message-ID: <672599.20287.qm@web33505.mail.mud.yahoo.com> Hi I am working on optimizing my code. one of the functions that I used in my code is MPI_ALLREDUCE. I want to have overlap between this communcation and my computation part of my code. is there any way to work around it? (it would be great to have sth like MPI_IALLREDUCE,MPI_WAIT) Mehdi --------------------------------- Get the Yahoo! toolbar and be alerted to new email wherever you're surfing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aja2111 at columbia.edu Tue Jul 10 19:48:37 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Tue, 10 Jul 2007 20:48:37 -0400 Subject: non collective mpi_allreduce In-Reply-To: <672599.20287.qm@web33505.mail.mud.yahoo.com> References: <672599.20287.qm@web33505.mail.mud.yahoo.com> Message-ID: <37604ab40707101748j3667dbya9536e9630ff2146@mail.gmail.com> The term you are looking for is non-blocking, a non-collective reduce is almost an oxymoron. And no, non-blocking reduces are not anywhere in the MPI Standard, maybe one of these days. Your best bet is to write an implementation yourself using MPI_ISEND and a tree structure which takes advantage of your network topology. This isn't exactly a question for PETSc though, you might have better luck on the mpich-users mailing list. You could also look in the literature to see how people are implementing global asynchronous codes cleverly. Let me know what you find (feel free to respond to aja2111 at columbia.edu), I'm interested in this sort of work as well. ~A On 7/10/07, Mehdi Bostandoost wrote: > Hi > I am working on optimizing my code. one of the functions that I used in my > code is MPI_ALLREDUCE. I want to have overlap between this communcation and > my computation part of my code. is there any way to work around it? > (it would be great to have sth like MPI_IALLREDUCE,MPI_WAIT) > > Mehdi > > > > ________________________________ > Get the Yahoo! toolbar and be alerted to new email wherever you're surfing. > > From mbostandoust at yahoo.com Tue Jul 10 22:21:01 2007 From: mbostandoust at yahoo.com (Mehdi Bostandoost) Date: Tue, 10 Jul 2007 20:21:01 -0700 (PDT) Subject: non collective mpi_allreduce In-Reply-To: <37604ab40707101748j3667dbya9536e9630ff2146@mail.gmail.com> Message-ID: <189231.70712.qm@web33504.mail.mud.yahoo.com> Hi Aron thanks for your comments. I found the MPI_IALLREDUCE (http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.pe.doc/pe_linux42/am107l0016.html ) but it seems it is the mpi implementation by IBM. it would be great to have the same thing in mpich. Aron Ahmadia wrote: The term you are looking for is non-blocking, a non-collective reduce is almost an oxymoron. And no, non-blocking reduces are not anywhere in the MPI Standard, maybe one of these days. Your best bet is to write an implementation yourself using MPI_ISEND and a tree structure which takes advantage of your network topology. This isn't exactly a question for PETSc though, you might have better luck on the mpich-users mailing list. You could also look in the literature to see how people are implementing global asynchronous codes cleverly. Let me know what you find (feel free to respond to aja2111 at columbia.edu), I'm interested in this sort of work as well. ~A On 7/10/07, Mehdi Bostandoost wrote: > Hi > I am working on optimizing my code. one of the functions that I used in my > code is MPI_ALLREDUCE. I want to have overlap between this communcation and > my computation part of my code. is there any way to work around it? > (it would be great to have sth like MPI_IALLREDUCE,MPI_WAIT) > > Mehdi > > > > ________________________________ > Get the Yahoo! toolbar and be alerted to new email wherever you're surfing. > > --------------------------------- Pinpoint customers who are looking for what you sell. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jul 10 22:44:22 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 10 Jul 2007 22:44:22 -0500 (CDT) Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On Tue, 10 Jul 2007, Brian Biskeborn wrote: > > Hello, > > Does anyone here have experience with using Petsc on a Blue Gene system? > I'm at IBM's Almaden Research Center and am currently involved in porting > some code that uses Petsc to Blue Gene. I've managed to compile and run > the code, and it seems to produce the correct results, but it generates > many floating point alignment exceptions at runtime. Can you send a log of these messages? Is this on BGL or BGP? Does the program abort? [on encountering these messages] With the minimal runs I've done on BGL - I don't remember seing any such messages. > I traced some of these warnings to a call to MatAssemblyBegin, so > I'm thinking maybe the problem is in Petsc. The Blue Gene > architecture requires 8- or 16-byte memory alignment for floating > point operations, depending on whether they are issued in parallel, > so I tried bumping up the alignment value in > src/sys/src/memory/mal.c. This had no effect on the problem. [Barry can confirm this] the code in mal.c attempts to make sure the memory allocated by PETSc is aligned properly. [8 byte boundary for doubles] One possibility is that the data passed in to MatAssemblyBegin() is not aligned? Satish > I didn't write any of the code I'm porting, and I'm not too familiar with > it yet, so I suppose the problem could really be anywhere. My question > here is simply this: has anyone seen this problem? Does it sound like an > issue with Petsc, or is it more likely to be in the top-level code? > > Thanks, and best regards, > Brian > > From balay at mcs.anl.gov Tue Jul 10 22:45:54 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 10 Jul 2007 22:45:54 -0500 (CDT) Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On Tue, 10 Jul 2007, Satish Balay wrote: > On Tue, 10 Jul 2007, Brian Biskeborn wrote: > > > > > Hello, > > > > Does anyone here have experience with using Petsc on a Blue Gene system? > > I'm at IBM's Almaden Research Center and am currently involved in porting > > some code that uses Petsc to Blue Gene. I've managed to compile and run > > the code, and it seems to produce the correct results, but it generates > > many floating point alignment exceptions at runtime. > > Can you send a log of these messages? Is this on BGL or BGP? Does the > program abort? [on encountering these messages] > > With the minimal runs I've done on BGL - I don't remember seing any > such messages. > > > I traced some of these warnings to a call to MatAssemblyBegin, so > > I'm thinking maybe the problem is in Petsc. The Blue Gene > > architecture requires 8- or 16-byte memory alignment for floating > > point operations, depending on whether they are issued in parallel, > > so I tried bumping up the alignment value in > > src/sys/src/memory/mal.c. This had no effect on the problem. > > [Barry can confirm this] the code in mal.c attempts to make sure the > memory allocated by PETSc is aligned properly. [8 byte boundary for > doubles] > > One possibility is that the data passed in to MatAssemblyBegin() is > not aligned? Sorry - I meant to say MatSetValues().. Satish > > Satish > > > I didn't write any of the code I'm porting, and I'm not too familiar with > > it yet, so I suppose the problem could really be anywhere. My question > > here is simply this: has anyone seen this problem? Does it sound like an > > issue with Petsc, or is it more likely to be in the top-level code? > > > > Thanks, and best regards, > > Brian > > > > > > From bbiskebo at us.ibm.com Wed Jul 11 10:55:51 2007 From: bbiskebo at us.ibm.com (Brian Biskeborn) Date: Wed, 11 Jul 2007 08:55:51 -0700 Subject: Petsc on Blue Gene In-Reply-To: Message-ID: > Can you send a log of these messages? Is this on BGL or BGP? Does the > program abort? [on encountering these messages] The program does not abort on exceptions - the only evidence of the problem is messages in the event log reading "Kernel detected X floating point alignment exceptions" (where X is a number usually on the order of 10^5) followed by what looks like a series of register values. I'm running on BGL. > With the minimal runs I've done on BGL - I don't remember seing any > such messages. > [Barry can confirm this] the code in mal.c attempts to make sure the > memory allocated by PETSc is aligned properly. [8 byte boundary for > doubles] > One possibility is that the data passed in to MatAssemblyBegin() is > not aligned? This says to me that the unaligned data is probably being generated outside of Petsc. Thanks for the info, I now have a much better idea about where to look for the problem! Brian From balay at mcs.anl.gov Wed Jul 11 11:34:33 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 11 Jul 2007 11:34:33 -0500 (CDT) Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On Wed, 11 Jul 2007, Brian Biskeborn wrote: > > Can you send a log of these messages? Is this on BGL or BGP? Does the > > program abort? [on encountering these messages] > > The program does not abort on exceptions - the only evidence of the problem > is messages in the event log reading "Kernel detected X floating point > alignment exceptions" (where X is a number usually on the order of 10^5) > followed by what looks like a series of register values. I'm running on > BGL. Is this event log in some system logs that users have no access to? Where is this logfile? [I'm guessing its neither JOBID.output nor JOBID.error] > > > With the minimal runs I've done on BGL - I don't remember seing any > > such messages. > > > [Barry can confirm this] the code in mal.c attempts to make sure the > > memory allocated by PETSc is aligned properly. [8 byte boundary for > > doubles] > > > One possibility is that the data passed in to MatAssemblyBegin() is > > not aligned? > > This says to me that the unaligned data is probably being generated outside > of Petsc. Thanks for the info, I now have a much better idea about where to > look for the problem! If the problem exists in PETSc, it should be reporduceable with a PETSc example [perhaps mat/examples/tests/ex2.c - which does MatSetValues()] cqsub -n 2 -t 2 ex9 Satish From bbiskebo at us.ibm.com Wed Jul 11 13:26:36 2007 From: bbiskebo at us.ibm.com (Brian Biskeborn) Date: Wed, 11 Jul 2007 11:26:36 -0700 Subject: Petsc on Blue Gene In-Reply-To: Message-ID: > > > Can you send a log of these messages? Is this on BGL or BGP? Does the > > > program abort? [on encountering these messages] > > > > The program does not abort on exceptions - the only evidence of the problem > > is messages in the event log reading "Kernel detected X floating point > > alignment exceptions" (where X is a number usually on the order of 10^5) > > followed by what looks like a series of register values. I'm running on > > BGL. > Is this event log in some system logs that users have no access to? > Where is this logfile? [I'm guessing its neither JOBID.output nor > JOBID.error] The exceptions are displayed in the so-called RAS event log. Users here have direct access to this log, but I'm not sure the same facility is necessarily available at every Blue Gene installation. > > > > > With the minimal runs I've done on BGL - I don't remember seing any > > > such messages. > > > > > [Barry can confirm this] the code in mal.c attempts to make sure the > > > memory allocated by PETSc is aligned properly. [8 byte boundary for > > > doubles] > > > > > One possibility is that the data passed in to MatAssemblyBegin() is > > > not aligned? > > > > This says to me that the unaligned data is probably being generated outside > > of Petsc. Thanks for the info, I now have a much better idea about where to > > look for the problem! > If the problem exists in PETSc, it should be reporduceable with a > PETSc example [perhaps mat/examples/tests/ex2.c - which does > MatSetValues()] > cqsub -n 2 -t 2 ex9 Well, what do you know? I'm getting floating point exceptions with mat/examples/tests/ex2. There are two relevant lines in the RAS event log. They are: Kernel detected 6 floating point alignment exceptions (1) iar 0x0026f4a8, dear 0x0069c22c (2) iar 0x0026f4a8, dear 0x0069c23c (3) iar 0x0026f4a8, dear 0x0069c24c (4) iar 0x0026f4a8, dear 0x0069c25c (5) iar 0x0026f4a8, dear 0x0069c26c (6) iar 0x0026f4a8, dear 0x0069c27c Kernel detected 36 floating point alignment exceptions (29) iar 0x0026f4a8, dear 0x006b48ac (30) iar 0x0026f4a8, dear 0x006b48bc (31) iar 0x0026f4a8, dear 0x006b48cc (32) iar 0x0026f4a8, dear 0x006b48dc (33) iar 0x0026f4a8, dear 0x006b48ec (34) iar 0x0026f4a8, dear 0x006b48fc (35) iar 0x0026f4a8, dear 0x006b490c (36) iar 0x0026f4a8, dear 0x006b491c The 42 exceptions above break down as follows: 10 in the test of MatNorm, 12 in MatTranspose, 10 in the 2nd MatNorm, and 10 during the test of MatAXPY. I compiled the Petsc I used above with a modified version of mal.c that #defines PETSC_MEMALIGN to 32 (to be on the safe side) and uses the posix_memalign call to allocate aligned memory. As far as I know, the only way to produce alignment exceptions would be to manually produce a misalignment somewhere (for example, storing an int immediately followed by a double at the beginning of an aligned memory block). Sorry, I totally forgot to mention this earlier: the code I'm working with requires Petsc 2.3.0, so I'm not using the latest version. Any suggestions on where this data misalignment might be occurring? Thanks, Brian From balay at mcs.anl.gov Wed Jul 11 13:53:24 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 11 Jul 2007 13:53:24 -0500 (CDT) Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On Wed, 11 Jul 2007, Brian Biskeborn wrote: > Well, what do you know? I'm getting floating point exceptions with > mat/examples/tests/ex2. There are two relevant lines in the RAS event log. > They are: > Kernel detected 6 floating point alignment exceptions (1) iar 0x0026f4a8, > dear 0x0069c22c (2) iar 0x0026f4a8, dear 0x0069c23c (3) iar 0x0026f4a8, > dear 0x0069c24c (4) iar 0x0026f4a8, dear 0x0069c25c (5) iar 0x0026f4a8, > dear 0x0069c26c (6) iar 0x0026f4a8, dear 0x0069c27c > Kernel detected 36 floating point alignment exceptions (29) iar 0x0026f4a8, > dear 0x006b48ac (30) iar 0x0026f4a8, dear 0x006b48bc (31) iar 0x0026f4a8, > dear 0x006b48cc (32) iar 0x0026f4a8, dear 0x006b48dc (33) iar 0x0026f4a8, > dear 0x006b48ec (34) iar 0x0026f4a8, dear 0x006b48fc (35) iar 0x0026f4a8, > dear 0x006b490c (36) iar 0x0026f4a8, dear 0x006b491c > > The 42 exceptions above break down as follows: 10 in the test of MatNorm, > 12 in MatTranspose, 10 in the 2nd MatNorm, and 10 during the test of > MatAXPY. How do you know the location of these exceptions? Can you narrow down further to the correct function name/source line? Also do you use --with-debugging=0 for this build? Do you get the smae errors wih '--with-debugging=1' build? > I compiled the Petsc I used above with a modified version of mal.c that > #defines PETSC_MEMALIGN to 32 (to be on the safe side) and uses the > posix_memalign call to allocate aligned memory. As far as I know, the only > way to produce alignment exceptions would be to manually produce a > misalignment somewhere (for example, storing an int immediately followed by > a double at the beginning of an aligned memory block). > > Sorry, I totally forgot to mention this earlier: the code I'm working with > requires Petsc 2.3.0, so I'm not using the latest version. > > Any suggestions on where this data misalignment might be occurring? Currently I have no clue as to where these alignment errors might come from. Satish From dalcinl at gmail.com Wed Jul 11 15:54:36 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Jul 2007 17:54:36 -0300 Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On 7/11/07, Brian Biskeborn wrote: > Any suggestions on where this data misalignment might be occurring? Please, take into account my ignorance about this is really complete, but.. supporte you declare a local variable and next take a pointer to it, ie: PetscReal rval, *rptr; rptr = &rval; Is the rptr aligned as you need? If that's not the case, perhaps PETSc is interacting bad with malloced memory and 'stacked' memory. Please, forget me if this does not make any sense. Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bbiskebo at us.ibm.com Wed Jul 11 19:25:39 2007 From: bbiskebo at us.ibm.com (Brian Biskeborn) Date: Wed, 11 Jul 2007 17:25:39 -0700 Subject: Petsc on Blue Gene In-Reply-To: Message-ID: > How do you know the location of these exceptions? Can you narrow down further > to the correct function name/source line? I found the locations of the exceptions by forcing an abort at various places in the code and counting the exceptions. Line 62 (counting from 1) of ex2.c generates 10 errors: ierr = MatView(mat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); Line 71 generates 12 errors: ierr = MatTranspose(mat,&tmat);CHKERRQ(ierr);; Line 82 generates 10 errors: ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); Line 91 generates 10 errors: ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); So the exceptions are occurring in MatView and MatTranspose here. > Also do you use --with-debugging=0 for this build? Do you get the smae > errors wih '--with-debugging=1' build? I've been running with debugging=0, but the same errors occur with debugging=1. I have also improved my understanding of Blue Gene's alignment requirements: experimentally, it looks like double values must be 4-byte aligned, but they cannot cross a 16-byte boundary. That is, the address of a double must be 0, 4, or 8 modulo 16. So if everything is indeed 8-byte aligned, there should be no problem. Lisandro: The compiler guarantees proper alignment of stack-allocated and statically-allocated data. Also, I think the Blue Gene implementation of malloc always returns 16-byte aligned addresses. That means the only way to get floating point exceptions is to use malloc'ed memory in such a way that alignment is disrupted. Brian From knepley at gmail.com Wed Jul 11 19:47:53 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 11 Jul 2007 19:47:53 -0500 Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: 1) This is for petsc-maint at mcs.anl.gov 2) I do not understand these source locations. Obviously, nothing is going on on these lines. Can't we get an exception right at the access point for the double? Matt On 7/11/07, Brian Biskeborn wrote: > > How do you know the location of these exceptions? Can you narrow down > further > > to the correct function name/source line? > > I found the locations of the exceptions by forcing an abort at various > places in the code and counting the exceptions. > > Line 62 (counting from 1) of ex2.c generates 10 errors: > ierr = MatView(mat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > Line 71 generates 12 errors: > ierr = MatTranspose(mat,&tmat);CHKERRQ(ierr);; > > Line 82 generates 10 errors: > ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > Line 91 generates 10 errors: > ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > So the exceptions are occurring in MatView and MatTranspose here. > > > Also do you use --with-debugging=0 for this build? Do you get the smae > > errors wih '--with-debugging=1' build? > > I've been running with debugging=0, but the same errors occur with > debugging=1. > > I have also improved my understanding of Blue Gene's alignment > requirements: experimentally, it looks like double values must be 4-byte > aligned, but they cannot cross a 16-byte boundary. That is, the address of > a double must be 0, 4, or 8 modulo 16. So if everything is indeed 8-byte > aligned, there should be no problem. > > Lisandro: > The compiler guarantees proper alignment of stack-allocated and > statically-allocated data. Also, I think the Blue Gene implementation of > malloc always returns 16-byte aligned addresses. That means the only way to > get floating point exceptions is to use malloc'ed memory in such a way that > alignment is disrupted. > > Brian > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Wed Jul 11 20:07:07 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Jul 2007 22:07:07 -0300 Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: On 7/11/07, Brian Biskeborn wrote: > Lisandro: > The compiler guarantees proper alignment of stack-allocated and > statically-allocated data. Also, I think the Blue Gene implementation of > malloc always returns 16-byte aligned addresses. In my 32bits linux FC6 box, I compile the following #include #include typedef unsigned us; #define PRINTP(p) \ printf("p=%p p mod 4=%u p mod 8=%u \n",\ p, (us)(((us)p)%4), (us)(((us)p)%8)) int main(int argc, char* argv[]) { int iv1,iv2,iv3; int *ip1 = &iv1; int *ip2 = &iv2; int *ip3 = &iv3; printf("stacked\n"); PRINTP(ip1); PRINTP(ip2); PRINTP(ip3); ip1 = (int*) malloc(sizeof(int)); ip2 = (int*) malloc(sizeof(int)); ip3 = (int*) malloc(sizeof(int)); printf("malloced\n"); PRINTP(ip1); PRINTP(ip2); PRINTP(ip3); return 0; } and next run $ a.out stacked p=0xbff66184 p mod 4=0 p mod 8=4 p=0xbff66180 p mod 4=0 p mod 8=0 p=0xbff6617c p mod 4=0 p mod 8=4 malloced p=0x9caf008 p mod 4=0 p mod 8=0 p=0x9caf018 p mod 4=0 p mod 8=0 p=0x9caf028 p mod 4=0 p mod 8=0 So malloced mem is 8 bytes aligned (is this the definition for aligned?), but addresses to local, stacked integer variables are 4 bytes. So this makes me think that perhaps PETSc is treating all adresses as malloced and in some place this can cause problems (you said you were using a custom PETSC_MEMALIGN)... As I said, my knowledge of this is 0 (zero), this is just an idea, and forget me if all this does not make any sense -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bsmith at mcs.anl.gov Wed Jul 11 22:06:09 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 11 Jul 2007 22:06:09 -0500 (CDT) Subject: Petsc on Blue Gene In-Reply-To: References: Message-ID: PetscMalloc() is designed so that all mallocs are double alligned. PetscTrMallocDefault() calls PetscMallocAlign() in src/sys/memory/mal.c How could something go wrong? If PETSC_HAVE_DOUBLE_ALIGN_MALLOC is set but mallocs are not always double aligned there will be a problem. If the system call memalign() is broken. If there is a bug in our code that tries to manually align at 8 bytes. Barry On Wed, 11 Jul 2007, Brian Biskeborn wrote: > > How do you know the location of these exceptions? Can you narrow down > further > > to the correct function name/source line? > > I found the locations of the exceptions by forcing an abort at various > places in the code and counting the exceptions. > > Line 62 (counting from 1) of ex2.c generates 10 errors: > ierr = MatView(mat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > Line 71 generates 12 errors: > ierr = MatTranspose(mat,&tmat);CHKERRQ(ierr);; > > Line 82 generates 10 errors: > ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > Line 91 generates 10 errors: > ierr = MatView(tmat,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > So the exceptions are occurring in MatView and MatTranspose here. > > > Also do you use --with-debugging=0 for this build? Do you get the smae > > errors wih '--with-debugging=1' build? > > I've been running with debugging=0, but the same errors occur with > debugging=1. > > I have also improved my understanding of Blue Gene's alignment > requirements: experimentally, it looks like double values must be 4-byte > aligned, but they cannot cross a 16-byte boundary. That is, the address of > a double must be 0, 4, or 8 modulo 16. So if everything is indeed 8-byte > aligned, there should be no problem. > > Lisandro: > The compiler guarantees proper alignment of stack-allocated and > statically-allocated data. Also, I think the Blue Gene implementation of > malloc always returns 16-byte aligned addresses. That means the only way to > get floating point exceptions is to use malloc'ed memory in such a way that > alignment is disrupted. > > Brian > > From aja2111 at columbia.edu Fri Jul 13 09:18:09 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Fri, 13 Jul 2007 10:18:09 -0400 Subject: Windows version of PETSc In-Reply-To: <000001c7adb5$62c286f0$4598580a@persistent.co.in> References: <000001c7adb5$62c286f0$4598580a@persistent.co.in> Message-ID: <37604ab40707130718v361fefbq1a5421863ade92ad@mail.gmail.com> Hi Sumit, I've posted this message to petsc-users, which is the right mailing list for this sort of question. You want to head to the official PETSc page if you'd like to download the source: http://www-unix.mcs.anl.gov/petsc/petsc-2/download/index.html The installation instructions are also located there. Good luck, ~Aron On 6/13/07, Sumit Vaidya wrote: > > > > > Hi, > > > > These are the machines supported by PETSc. > > http://www.caspur.it/risorse/softappl/doc/petsc_docs/machines.html > > > > From this, I came to know that PETSc is already available for Windows with > 32-bit processor. > > Where can I get this software version of PETSc for windows? > > > > Waiting for your reply, > > Sumit > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Pvt. Ltd. It is > intended only for the use of the individual or entity to which it is > addressed. If you are not the intended recipient, you are not authorized to > read, retain, copy, print, distribute or use this message. If you have > received this communication in error, please notify the sender and delete > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. From jagruti.trivedi at navy.mil Mon Jul 16 10:16:32 2007 From: jagruti.trivedi at navy.mil (Trivedi, Jagruti CIV 470000D, 474300D) Date: Mon, 16 Jul 2007 08:16:32 -0700 Subject: Windows version of PETSc In-Reply-To: <000001c7adb5$62c286f0$4598580a@persistent.co.in> References: <000001c7adb5$62c286f0$4598580a@persistent.co.in> Message-ID: <7F49B2C8312BE84FBC6BE7349A7EC58F05492C21@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> Hi, Is PetSc supported for Solaris 10 for X86? The following link describe only Solaris 5.7 Or Sparcstation is supported. Response is appreciated. ...Jagruti -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Sumit Vaidya Sent: Wednesday, June 13, 2007 5:22 To: petsc-dev at mcs.anl.gov; petsc-users at mcs.anl.gov Cc: 'Sumit Vaidya' Subject: Windows version of PETSc Hi, These are the machines supported by PETSc. http://www.caspur.it/risorse/softappl/doc/petsc_docs/machines.html >From this, I came to know that PETSc is already available for Windows with 32-bit processor. Where can I get this software version of PETSc for windows? Waiting for your reply, Sumit DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. From balay at mcs.anl.gov Mon Jul 16 10:29:59 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 16 Jul 2007 10:29:59 -0500 (CDT) Subject: Windows version of PETSc In-Reply-To: <7F49B2C8312BE84FBC6BE7349A7EC58F05492C21@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> References: <000001c7adb5$62c286f0$4598580a@persistent.co.in> <7F49B2C8312BE84FBC6BE7349A7EC58F05492C21@NAWECHLKEX02VA.nadsuswe.nads.navy.mil> Message-ID: - could you avoid cross posting to multiple lists? [this issue is more relavent to petsc-users list] Our current testbox is: sparc SunOS 5.9. However it should work on any solaris versions. If you encounter problems, let us know at petsc-maint at mcs.anl.gov Satish On Mon, 16 Jul 2007, Trivedi, Jagruti CIV 470000D, 474300D wrote: > > Hi, > > Is PetSc supported for Solaris 10 for X86? The following link describe > only Solaris 5.7 > Or Sparcstation is supported. > > Response is appreciated. > > ...Jagruti > > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Sumit Vaidya > Sent: Wednesday, June 13, 2007 5:22 > To: petsc-dev at mcs.anl.gov; petsc-users at mcs.anl.gov > Cc: 'Sumit Vaidya' > Subject: Windows version of PETSc > > Hi, > > > > These are the machines supported by PETSc. > > http://www.caspur.it/risorse/softappl/doc/petsc_docs/machines.html > > > > From this, I came to know that PETSc is already available for Windows > with 32-bit processor. > > Where can I get this software version of PETSc for windows? > > > > Waiting for your reply, > > Sumit > > DISCLAIMER ========== This e-mail may contain privileged and > confidential information which is the property of Persistent Systems > Pvt. Ltd. It is intended only for the use of the individual or entity to > which it is addressed. If you are not the intended recipient, you are > not authorized to read, retain, copy, print, distribute or use this > message. If you have received this communication in error, please notify > the sender and delete all copies of this message. Persistent Systems > Pvt. Ltd. does not accept any liability for virus infected mails. > > From sumit_vaidya at persistent.co.in Thu Jul 19 09:38:50 2007 From: sumit_vaidya at persistent.co.in (Sumit Vaidya) Date: Thu, 19 Jul 2007 20:08:50 +0530 Subject: PETSc compiler Message-ID: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> Hi, Where is the option to change the compiler? I saw $PCC in the bmake/common/variables file. Now I want to change the compiler. Where should I specify it? Regards, Sumit DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jul 19 09:42:03 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 19 Jul 2007 09:42:03 -0500 (CDT) Subject: PETSc compiler In-Reply-To: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> Message-ID: On Thu, 19 Jul 2007, Sumit Vaidya wrote: > Where is the option to change the compiler? > > I saw $PCC in the bmake/common/variables file. Now I want to change the > compiler. > > > > Where should I specify it? The correct thing to do here is specify the compilers to configure. ./config/configure.py --with-cc=gcc --with-fc=g77 --download-mpich=1 --download-f-blas-lapack=1 If you need to use PETSc from different compilers [or other variation in options] create multiple installs of PETSc with different values of PETSC_ARCH ./config/configure.py --with-cc=gcc PETSC_ARCH=linux-gcc make PETSC_ARCH=linux-gcc ./config/configure.py --with-cc=icc PETSC_ARCH=linux-icc make PETSC_ARCH=linux-icc etc.. Satish From knepley at gmail.com Thu Jul 19 09:42:11 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 19 Jul 2007 09:42:11 -0500 Subject: PETSc compiler In-Reply-To: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> Message-ID: On 7/19/07, Sumit Vaidya wrote: > Hi, > > > > Where is the option to change the compiler? There is documentation on the installation page: http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html and there is also documentation of all the options to configure: ./config/configure --help The option you want is --with-cc. Thanks, Matt > I saw $PCC in the bmake/common/variables file. Now I want to change the > compiler. > > > > Where should I specify it? > > > > Regards, > > Sumit > > > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Pvt. Ltd. It is > intended only for the use of the individual or entity to which it is > addressed. If you are not the intended recipient, you are not authorized to > read, retain, copy, print, distribute or use this message. If you have > received this communication in error, please notify the sender and delete > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From aja2111 at columbia.edu Thu Jul 19 09:52:04 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Thu, 19 Jul 2007 10:52:04 -0400 Subject: PETSc compiler In-Reply-To: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> Message-ID: <37604ab40707190752n4ec169cem73dcf6e20bbac9f@mail.gmail.com> Sumit, The compilers are specified at the command-line when you make your configure call. i.e. from $PETSC_DIR ./config/configure.py --with-cc=gcc if you need more information, try ./config/configure.py --help And to answer your question, the compiler to use is stored in a configuration file specific to your installation (generated by configure.py), not in the generic common files. Cheers ~A On 7/19/07, Sumit Vaidya wrote: > > > > > Hi, > > > > Where is the option to change the compiler? > > I saw $PCC in the bmake/common/variables file. Now I want to change the > compiler. > > > > Where should I specify it? > > > > Regards, > > Sumit > > > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Pvt. Ltd. It is > intended only for the use of the individual or entity to which it is > addressed. If you are not the intended recipient, you are not authorized to > read, retain, copy, print, distribute or use this message. If you have > received this communication in error, please notify the sender and delete > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. From sumit_vaidya at persistent.co.in Thu Jul 19 10:03:06 2007 From: sumit_vaidya at persistent.co.in (Sumit Vaidya) Date: Thu, 19 Jul 2007 20:33:06 +0530 Subject: PETSc compiler In-Reply-To: References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> Message-ID: <000001c7ca15$e9be3830$4598580a@persistent.co.in> I am sorry. I want to build it on windows. I forgot to mention that. On windows I will not be having "configure.py" script. Regards, Sumit -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, July 19, 2007 8:12 PM To: petsc-users at mcs.anl.gov Subject: Re: PETSc compiler On Thu, 19 Jul 2007, Sumit Vaidya wrote: > Where is the option to change the compiler? > > I saw $PCC in the bmake/common/variables file. Now I want to change the > compiler. > > > > Where should I specify it? The correct thing to do here is specify the compilers to configure. ./config/configure.py --with-cc=gcc --with-fc=g77 --download-mpich=1 --download-f-blas-lapack=1 If you need to use PETSc from different compilers [or other variation in options] create multiple installs of PETSc with different values of PETSC_ARCH ./config/configure.py --with-cc=gcc PETSC_ARCH=linux-gcc make PETSC_ARCH=linux-gcc ./config/configure.py --with-cc=icc PETSC_ARCH=linux-icc make PETSC_ARCH=linux-icc etc.. Satish DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. From balay at mcs.anl.gov Thu Jul 19 10:05:12 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 19 Jul 2007 10:05:12 -0500 (CDT) Subject: PETSc compiler In-Reply-To: <000001c7ca15$e9be3830$4598580a@persistent.co.in> References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> <000001c7ca15$e9be3830$4598580a@persistent.co.in> Message-ID: On Thu, 19 Jul 2007, Sumit Vaidya wrote: > I am sorry. > > I want to build it on windows. I forgot to mention that. On windows I will > not be having "configure.py" script. You'll have to follow the installation instructions from: http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html you'll need to run config/configure.py [from cygwin] on windows - as per the installation instructions. Sepecifically - check the instruction in 'Additional Microsoft Windows Notes:' section. BTW: what compilers [c, fortran] do you plan to use on windows? Satish From gtg100n at mail.gatech.edu Thu Jul 19 16:05:16 2007 From: gtg100n at mail.gatech.edu (Alejandro Garzon) Date: Thu, 19 Jul 2007 17:05:16 -0400 Subject: function for adding matrices Message-ID: <1184879116.469fd20c138c8@webmail.mail.gatech.edu> Hi, is there a function in petsc for adding two matrices? or one for multiplying a matrix by a scalar? I looked several times in the manual pages and couldn't find any. If they don't exist, why is it so? Thanks. -- Alejandro From knepley at gmail.com Thu Jul 19 16:08:16 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 19 Jul 2007 16:08:16 -0500 Subject: function for adding matrices In-Reply-To: <1184879116.469fd20c138c8@webmail.mail.gatech.edu> References: <1184879116.469fd20c138c8@webmail.mail.gatech.edu> Message-ID: On 7/19/07, Alejandro Garzon wrote: > Hi, is there a function in petsc for adding two matrices? or one for multiplying http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatAXPY.html > a matrix by a scalar? I looked several times in the manual pages and couldn't http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatScale.html Thanks, Matt > find any. If they don't exist, why is it so? Thanks. > -- > Alejandro > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From sumit_vaidya at persistent.co.in Fri Jul 20 00:16:25 2007 From: sumit_vaidya at persistent.co.in (Sumit Vaidya) Date: Fri, 20 Jul 2007 10:46:25 +0530 Subject: PETSc compiler In-Reply-To: References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> <000001c7ca15$e9be3830$4598580a@persistent.co.in> Message-ID: <000301c7ca8d$1eaf1a60$4598580a@persistent.co.in> Well I am planning to use icl and ifort compilers on windows. Is it possible to build it without Cygwin? Sumit -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, July 19, 2007 8:35 PM To: petsc-users at mcs.anl.gov Subject: RE: PETSc compiler On Thu, 19 Jul 2007, Sumit Vaidya wrote: > I am sorry. > > I want to build it on windows. I forgot to mention that. On windows I will > not be having "configure.py" script. You'll have to follow the installation instructions from: http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html you'll need to run config/configure.py [from cygwin] on windows - as per the installation instructions. Sepecifically - check the instruction in 'Additional Microsoft Windows Notes:' section. BTW: what compilers [c, fortran] do you plan to use on windows? Satish DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. From bsmith at mcs.anl.gov Fri Jul 20 07:57:26 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 20 Jul 2007 07:57:26 -0500 (CDT) Subject: PETSc compiler In-Reply-To: <000301c7ca8d$1eaf1a60$4598580a@persistent.co.in> References: <000001c7ca12$85d0bc60$4598580a@persistent.co.in> <000001c7ca15$e9be3830$4598580a@persistent.co.in> <000301c7ca8d$1eaf1a60$4598580a@persistent.co.in> Message-ID: No, the cygwin shell and its make are used to configure and compile all of the PETSc libraries. Once the libraries are built you can switch to Developers Studio or what you want to use to build your application. Barry On Fri, 20 Jul 2007, Sumit Vaidya wrote: > Well I am planning to use icl and ifort compilers on windows. > > Is it possible to build it without Cygwin? > > Sumit > > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Satish Balay > Sent: Thursday, July 19, 2007 8:35 PM > To: petsc-users at mcs.anl.gov > Subject: RE: PETSc compiler > > On Thu, 19 Jul 2007, Sumit Vaidya wrote: > > > I am sorry. > > > > I want to build it on windows. I forgot to mention that. On windows I will > > not be having "configure.py" script. > > You'll have to follow the installation instructions from: > http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html > > you'll need to run config/configure.py [from cygwin] on windows - > as per the installation instructions. > > Sepecifically - check the instruction in 'Additional Microsoft Windows > Notes:' section. > > BTW: what compilers [c, fortran] do you plan to use on windows? > > Satish > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. > > From tim at cevis.uni-bremen.de Mon Jul 23 03:08:34 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Mon, 23 Jul 2007 10:08:34 +0200 (CEST) Subject: Read matrix Message-ID: Dear all, I am assembling a matrix using several calls to MatSetValues(). Then, I call MatAssemblyBegin() and MatAssemblyEnd() as usual. After that, I would like to write the complete matrix into a file (for debugging purposes). My idea was to call MatGetRow() for each matrix row and write the values out. This seems to work in one-processor mode, but in parallel, I get [14]PETSC ERROR: MatGetRow_MPIAIJ() line 1185 in src/mat/impls/aij/mpi/mpiaij.c [14]PETSC ERROR: Object is in wrong state! [14]PETSC ERROR: Already active! [14]PETSC ERROR: MatGetRow() line 106 in src/mat/interface/matrix.c What did I do wrong? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From sumit_vaidya at persistent.co.in Mon Jul 23 03:53:09 2007 From: sumit_vaidya at persistent.co.in (Sumit Vaidya) Date: Mon, 23 Jul 2007 14:23:09 +0530 Subject: Configuration crash on Windows : ALWAYS Message-ID: <000001c7cd06$e4c1c330$4598580a@persistent.co.in> When I build PETSc 2.3.3-p3 on Windows I always get the following error. Exception in thread Shell Command: Traceback (most recent call last): File "C:\Python\lib\threading.py", line 460, in __bootstrap self.run() File "C\Python\lib\threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) And I see "CONFIGURATION CRASH". This is always. Reading about this on the mailing lists, I came to know that it is related to some test case. Is there any way to stop this testing of library? How to resolve this error? Waiting for your reply, Sumit DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knutert at stud.ntnu.no Mon Jul 23 03:29:56 2007 From: knutert at stud.ntnu.no (Knut Erik Teigen) Date: Mon, 23 Jul 2007 10:29:56 +0200 Subject: Read matrix In-Reply-To: References: Message-ID: <1185179396.12643.43.camel@iept0415.ivt.ntnu.no> Hello, Can't you just use MatView, with a file writer viewer? You can save to both ASCII and binary files, using PetscViewerASCIIOpen and PetscViewerBinaryOpen. Regards, Knut Erik Teigen On Mon, 2007-07-23 at 10:08 +0200, Tim Kr?ger wrote: > Dear all, > > I am assembling a matrix using several calls to MatSetValues(). > Then, I call MatAssemblyBegin() and MatAssemblyEnd() as usual. After > that, I would like to write the complete matrix into a file (for > debugging purposes). My idea was to call MatGetRow() for each matrix > row and write the values out. This seems to work in one-processor > mode, but in parallel, I get > > [14]PETSC ERROR: MatGetRow_MPIAIJ() line 1185 in src/mat/impls/aij/mpi/mpiaij.c > [14]PETSC ERROR: Object is in wrong state! > [14]PETSC ERROR: Already active! > [14]PETSC ERROR: MatGetRow() line 106 in src/mat/interface/matrix.c > > What did I do wrong? > > Best Regards, > > Tim > From tim at cevis.uni-bremen.de Mon Jul 23 04:23:28 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Mon, 23 Jul 2007 11:23:28 +0200 (CEST) Subject: Read matrix In-Reply-To: <1185179396.12643.43.camel@iept0415.ivt.ntnu.no> References: <1185179396.12643.43.camel@iept0415.ivt.ntnu.no> Message-ID: Dear Knut, On Mon, 23 Jul 2007, Knut Erik Teigen wrote: > Can't you just use MatView, with a file writer viewer? You can save to > both ASCII and binary files, using PetscViewerASCIIOpen and > PetscViewerBinaryOpen. Thank you for your answer. I did not know the MatView() function. Anyway, I found out what I did wrong: I forgot to call MatRestoreRow() each time. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From tim at cevis.uni-bremen.de Mon Jul 23 04:41:40 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Mon, 23 Jul 2007 11:41:40 +0200 (CEST) Subject: KSP/PC choice Message-ID: Dear all, the more KSP and PC methods are available, the more difficult it is to choose a suitable one. I want to compute Navier-Stokes (using Finite Elements) for medium-sized Reynolds numbers (still laminar, though). I am currently using -ksp_type gmres -ksp_gmres_restart 30 -pc_type bjacobi -sub_pc_type asm -sub_pc_asm_overlap 2 -sub_sub_pc_type ilu -sub_sub_pc_factor_zeropivot 0 -sub_sub_pc_ilu_levels 1 but with a large number of CPUs (e.g. 20), no convergence is achieved. Does anyone have some suggestions for a better setting? I am using petsc-2.3.0. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From gaatenek at irisa.fr Mon Jul 23 05:33:09 2007 From: gaatenek at irisa.fr (gaatenek at irisa.fr) Date: Mon, 23 Jul 2007 12:33:09 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: Message-ID: <2999.131.254.11.48.1185186789.squirrel@mail.irisa.fr> Dear Tims, Do you monitoring the convergence? Try it a see if you have stagnation in your convergence. However you can try this setting, -ksp_type gmres -ksp_gmres_restart 30 -pc_type asm -pc_asm_overlap 2 -sub_pc_type ilu -sub_pc_factor_zeropivot 0 -sub_pc_ilu_levels 3 Regards, > Dear all, > > the more KSP and PC methods are available, the more difficult it is to > choose a suitable one. > > I want to compute Navier-Stokes (using Finite Elements) for > medium-sized Reynolds numbers (still laminar, though). I am currently > using > > -ksp_type gmres > -ksp_gmres_restart 30 > -pc_type bjacobi > -sub_pc_type asm > -sub_pc_asm_overlap 2 > -sub_sub_pc_type ilu > -sub_sub_pc_factor_zeropivot 0 > -sub_sub_pc_ilu_levels 1 > > but with a large number of CPUs (e.g. 20), no convergence is achieved. > Does anyone have some suggestions for a better setting? > > I am using petsc-2.3.0. > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > From saut.olivier at wanadoo.fr Mon Jul 23 06:47:38 2007 From: saut.olivier at wanadoo.fr (Olivier Saut) Date: Mon, 23 Jul 2007 13:47:38 +0200 Subject: DAVec and MPI Gather Message-ID: Hi all, something is not very clear for me. I use a DA to solve a diffusion equation on several processors. The global vector containing the solution is created with DACreateGlobalVector(da,&Sol); VecDuplicate(Sol, sMemb); // RHS vector // Local part of the matrix and RHS vector are computed KSPSolve(ksp, sMemb, Sol); At this stage, Sol is supposed to contained the solution. Now for vizualisation, I need to get acces all the values of Sol. As far as I understand, DAVecGetArray only gives acces to the local values of Sol. Should I use MPI Gather to obtain the values from the other processors? Thanks a lot, - Olivier From dalcinl at gmail.com Mon Jul 23 09:20:42 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 23 Jul 2007 11:20:42 -0300 Subject: KSP/PC choice In-Reply-To: References: Message-ID: On 7/23/07, Tim Kr?ger wrote: > Dear all, > > the more KSP and PC methods are available, the more difficult it is to > choose a suitable one. > > I want to compute Navier-Stokes (using Finite Elements) for > medium-sized Reynolds numbers (still laminar, though). I am currently > using Are you solving incompresible flow? Tansient or steady state? Are you using a monolithic formulation; or a fractional-step like formulation? What kind of stabilization for advection and pressure are you using? I ask this because in my experience incompressible flows with monolithic formulations are usually hard to solve, and a good choice of stabilization method not only leads to better final solution, but also lead to linear systems with better conditioning. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From timothy.stitt at ichec.ie Mon Jul 23 09:24:07 2007 From: timothy.stitt at ichec.ie (Tim Stitt) Date: Mon, 23 Jul 2007 15:24:07 +0100 Subject: Getting Remote Distributed Vector Elements? Message-ID: <200707231524.07705.timothy.stitt@ichec.ie> Hi all, I was just wondering if anyone can suggest the most efficient method for obtaining any given element in a distributed vector? If I understand correctly VecGetValues() only returns local elements? I need to access non-local elements also. What is the best approach? Thanks in advance, Tim. -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From tim at cevis.uni-bremen.de Mon Jul 23 09:26:13 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Mon, 23 Jul 2007 16:26:13 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: <2999.131.254.11.48.1185186789.squirrel@mail.irisa.fr> References: <2999.131.254.11.48.1185186789.squirrel@mail.irisa.fr> Message-ID: Dear gaatenek On Mon, 23 Jul 2007, gaatenek at irisa.fr wrote: > Do you monitoring the convergence? Try it a see if you have stagnation in > your convergence. Yes, I monitor it, but that doesn't seem to give me any helpful information -- except that it stagnates of course. > However you can try this setting, > -ksp_type gmres > -ksp_gmres_restart 30 > -pc_type asm > -pc_asm_overlap 2 > -sub_pc_type ilu > -sub_pc_factor_zeropivot 0 > -sub_pc_ilu_levels 3 Thank you, I tried it, and it worked well for a test case. However, for the real application, the PC eats up all my memory (even in cases where my former setting did work). Any hints (except buying a larger computer)? Might some sub_ksp method help? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From knepley at gmail.com Mon Jul 23 09:39:29 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Jul 2007 09:39:29 -0500 Subject: Getting Remote Distributed Vector Elements? In-Reply-To: <200707231524.07705.timothy.stitt@ichec.ie> References: <200707231524.07705.timothy.stitt@ichec.ie> Message-ID: On 7/23/07, Tim Stitt wrote: > Hi all, > > I was just wondering if anyone can suggest the most efficient method for > obtaining any given element in a distributed vector? If I understand > correctly VecGetValues() only returns local elements? I need to access > non-local elements also. What is the best approach? You create a VecScatter with these indices into a local vector, and then get the array from that. Matt > Thanks in advance, > > Tim. > > -- > Dr. Timothy Stitt > HPC Application Consultant - ICHEC (www.ichec.ie) > > Dublin Institute for Advanced Studies > 5 Merrion Square - Dublin 2 - Ireland > > +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From knepley at gmail.com Mon Jul 23 09:42:52 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Jul 2007 09:42:52 -0500 Subject: DAVec and MPI Gather In-Reply-To: References: Message-ID: On 7/23/07, Olivier Saut wrote: > Hi all, > > something is not very clear for me. > I use a DA to solve a diffusion equation on several processors. > > The global vector containing the solution is created with > DACreateGlobalVector(da,&Sol); > VecDuplicate(Sol, sMemb); // RHS vector > > // Local part of the matrix and RHS vector are computed > > KSPSolve(ksp, sMemb, Sol); > > > At this stage, Sol is supposed to contained the solution. > > Now for vizualisation, I need to get acces all the values of Sol. > As far as I understand, DAVecGetArray only gives acces to the local > values of Sol. > Should I use MPI Gather to obtain the values from the other processors? Collecting the values on a single process is fundamentally unscalable and seems to defeat the purpose of computing in parallel. However, there is a method to do it: http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html Matt > Thanks a lot, > > - Olivier -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From tim at cevis.uni-bremen.de Mon Jul 23 09:58:17 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Mon, 23 Jul 2007 16:58:17 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: Message-ID: Dear Lissandro, On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > On 7/23/07, Tim Kr?ger wrote: >> >> the more KSP and PC methods are available, the more difficult it is to >> choose a suitable one. >> >> I want to compute Navier-Stokes (using Finite Elements) for >> medium-sized Reynolds numbers (still laminar, though). I am currently >> using > > Are you solving incompresible flow? Yes. > Tansient or steady state? Transient. > Are you using a monolithic formulation; or a fractional-step like > formulation? Monolithic. (I am not quite sure about this word but I assume it means that I don't try to decouple the two equations.) > What kind of stabilization for advection and pressure are you using? Streamline diffusion (for advection). I am not aware of the requirement to stabilize the pressure as well. > I ask this because in my experience incompressible flows with > monolithic formulations are usually hard to solve, and a good choice > of stabilization method not only leads to better final solution, but > also lead to linear systems with better conditioning. Would you recommend to use a fractional step method? Would you recommend a different stabilization method? If you have any suggestion, please let me know. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From knepley at gmail.com Mon Jul 23 10:01:11 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Jul 2007 10:01:11 -0500 Subject: KSP/PC choice In-Reply-To: References: Message-ID: 1) Until you run out of memory, I would use sparse direct like MUMPS 2) After that, as long as you have the memory I would increase the GMRES vectors, say to 50 or 100. 3) After that I would try LGMRES which generally converges better on these problems. Matt On 7/23/07, Tim Kr?ger wrote: > Dear Lissandro, > > On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > > > On 7/23/07, Tim Kr?ger wrote: > >> > >> the more KSP and PC methods are available, the more difficult it is to > >> choose a suitable one. > >> > >> I want to compute Navier-Stokes (using Finite Elements) for > >> medium-sized Reynolds numbers (still laminar, though). I am currently > >> using > > > > Are you solving incompresible flow? > > Yes. > > > Tansient or steady state? > > Transient. > > > Are you using a monolithic formulation; or a fractional-step like > > formulation? > > Monolithic. (I am not quite sure about this word but I assume it > means that I don't try to decouple the two equations.) > > > What kind of stabilization for advection and pressure are you using? > > Streamline diffusion (for advection). I am not aware of the > requirement to stabilize the pressure as well. > > > I ask this because in my experience incompressible flows with > > monolithic formulations are usually hard to solve, and a good choice > > of stabilization method not only leads to better final solution, but > > also lead to linear systems with better conditioning. > > Would you recommend to use a fractional step method? Would you > recommend a different stabilization method? If you have any > suggestion, please let me know. > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From gaatenek at irisa.fr Mon Jul 23 10:12:16 2007 From: gaatenek at irisa.fr (gaatenek at irisa.fr) Date: Mon, 23 Jul 2007 17:12:16 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: <2999.131.254.11.48.1185186789.squirrel@mail.irisa.fr> Message-ID: <3725.131.254.11.48.1185203536.squirrel@mail.irisa.fr> Dear Tims, The problem is how to solve more accuretly the local problem in ASM. You can also use KSP to solve local problem. I think you can look how to do it with PCASMGetSubKSP(). Best Regards, Guy > Dear gaatenek > > On Mon, 23 Jul 2007, gaatenek at irisa.fr wrote: > >> Do you monitoring the convergence? Try it a see if you have stagnation >> in >> your convergence. > > Yes, I monitor it, but that doesn't seem to give me any helpful > information -- except that it stagnates of course. > >> However you can try this setting, >> -ksp_type gmres >> -ksp_gmres_restart 30 >> -pc_type asm >> -pc_asm_overlap 2 >> -sub_pc_type ilu >> -sub_pc_factor_zeropivot 0 >> -sub_pc_ilu_levels 3 > > Thank you, I tried it, and it worked well for a test case. However, > for the real application, the PC eats up all my memory (even in cases > where my former setting did work). > > Any hints (except buying a larger computer)? > > Might some sub_ksp method help? > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > From knepley at gmail.com Mon Jul 23 10:17:16 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Jul 2007 10:17:16 -0500 Subject: Configuration crash on Windows : ALWAYS In-Reply-To: <000001c7cd06$e4c1c330$4598580a@persistent.co.in> References: <000001c7cd06$e4c1c330$4598580a@persistent.co.in> Message-ID: You did not send configure.log. However, try running with --useThreads=0. Matt On 7/23/07, Sumit Vaidya wrote: > > > > > When I build PETSc 2.3.3-p3 on Windows I always get the following error. > > > > Exception in thread Shell Command: > > Traceback (most recent call last): File "C:\Python\lib\threading.py", line > 460, in __bootstrap > > self.run() > > File "C\Python\lib\threading.py", line 440, in run > > self.__target(*self.__args, **self.__kwargs) > > > > And I see "CONFIGURATION CRASH". > > This is always. > > > > Reading about this on the mailing lists, I came to know that it is related > to some test case. Is there any way to stop this testing of library? > > > > How to resolve this error? > > > > Waiting for your reply, > > Sumit > > > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Pvt. Ltd. It is > intended only for the use of the individual or entity to which it is > addressed. If you are not the intended recipient, you are not authorized to > read, retain, copy, print, distribute or use this message. If you have > received this communication in error, please notify the sender and delete > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From balay at mcs.anl.gov Mon Jul 23 10:20:54 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 23 Jul 2007 10:20:54 -0500 (CDT) Subject: Configuration crash on Windows : ALWAYS In-Reply-To: References: <000001c7cd06$e4c1c330$4598580a@persistent.co.in> Message-ID: > Traceback (most recent call last): File "C:\Python\lib\threading.py", line PETSc requires cygwin python as per installation instruction. [looks like you are using non-cygwin python - which won't work] Satish On Mon, 23 Jul 2007, Matthew Knepley wrote: > You did not send configure.log. However, try running with --useThreads=0. > > Matt > > On 7/23/07, Sumit Vaidya wrote: > > > > > > > > > > When I build PETSc 2.3.3-p3 on Windows I always get the following error. > > > > > > > > Exception in thread Shell Command: > > > > Traceback (most recent call last): File "C:\Python\lib\threading.py", line > > 460, in __bootstrap > > > > self.run() > > > > File "C\Python\lib\threading.py", line 440, in run > > > > self.__target(*self.__args, **self.__kwargs) > > > > > > > > And I see "CONFIGURATION CRASH". > > > > This is always. > > > > > > > > Reading about this on the mailing lists, I came to know that it is related > > to some test case. Is there any way to stop this testing of library? > > > > > > > > How to resolve this error? > > > > > > > > Waiting for your reply, > > > > Sumit > > > > > > > > DISCLAIMER ========== This e-mail may contain privileged and confidential > > information which is the property of Persistent Systems Pvt. Ltd. It is > > intended only for the use of the individual or entity to which it is > > addressed. If you are not the intended recipient, you are not authorized to > > read, retain, copy, print, distribute or use this message. If you have > > received this communication in error, please notify the sender and delete > > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > > liability for virus infected mails. > > > From dalcinl at gmail.com Mon Jul 23 11:30:31 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 23 Jul 2007 13:30:31 -0300 Subject: KSP/PC choice In-Reply-To: References: Message-ID: On 7/23/07, Tim Kr?ger wrote: > Monolithic. (I am not quite sure about this word but I assume it > means that I don't try to decouple the two equations.) That's the meaning for me (not sure if the word is completelly correct) > > What kind of stabilization for advection and pressure are you using? > > Streamline diffusion (for advection). I am not aware of the > requirement to stabilize the pressure as well. Well, you need to stabilize pressure if you have a non div-stable finite element space (this happens when using linear, equal-order FE for both velocity and pressure). Surely you are using a stable FE. > Would you recommend to use a fractional step method? No. In my opinion, (traditional) fractional step methods could be problematic in many ways. > Would you > recommend a different stabilization method? If you have any > suggestion, please let me know. I'm not exactly sure what do you understand for 'streamline difussion' (is rathre general). But we normally use non div-stable FE pairs, formulated with SUPG/PSPG (the first for advection, the second for pressure) stabilization, as proposed by Tezduyar in many publications. This method is residual-based (thus consistent). In particular, the stabilization parameters take into accout the time step and the compressibility contraint. This not only leads to good solutions, but also to far better conditioned linear systems. However, this perhaps will not solve all your problems. The linear system have a saddle-point nature, and this is the source of all trouble to solve it. Specialized preconditioners are usually needed, among them: * EBE/ CEBE: (clustered-)elemet-by-element, you can find references in works from Hughes, Tezduyar. I've never tried them, as they seems to require working at the element level. * Block preconditioners, as suggested by Elman, Kay, Loghin, Wathen. Those methods decouples the equations at the PC step. I am working on this, as convergence deteriorate with increasing Reynolds. * Finally, we have our on way. It is not completely scalable, but usually works. It is related to interating only on the degree of freedom on the inteface between processors. This is implemented in a completelly algebraic manner, but relies in using a matrix tipe MATIS. This code is included inside the distribution of my Python wrappers for PETSc (petsc4py), but is coded in C, so usable in other scenarios. I've never tried to include this is PETSc, because I am not sure how much it can help to general users (in theory, iteration on interface Schur complement should be in some way similar to ASM). If you are interested in trying the my solution, I can help you. But I cannot assert you will have good performance, specially in big 3D problems. Anyway, we are currently using it to simulate the flow around a racing car (Re=16e6) with about 600K nodes in 30 processors. Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From DOMI0002 at ntu.edu.sg Mon Jul 23 21:44:07 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Tue, 24 Jul 2007 10:44:07 +0800 Subject: Poisson Eqn in a concentric cube Message-ID: Hi, I would like to solve the Poisson equation (3D) in a concentric type of cube as in the attached figure using PETSc. The solution domain would be the shaded region only and it extends inside the cube. I would like to have suggestions on ways to form and assemble the matrix associated with the discretization. Also, is it a good practise to discretize the entire domain and blank the grid points which are within the cube not in the domain ? Regards, Dominic -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cube.jpg Type: image/jpeg Size: 22382 bytes Desc: cube.jpg URL: From li76pan at yahoo.com Tue Jul 24 01:19:36 2007 From: li76pan at yahoo.com (li pan) Date: Mon, 23 Jul 2007 23:19:36 -0700 (PDT) Subject: PETSc compiler In-Reply-To: Message-ID: <182703.14515.qm@web36804.mail.mud.yahoo.com> hi Barry, I also have interest about how to use petsc under visual studio. Can you explain a little bit how to do it? I'm using cygwin. Cygwin generates libraries for linux environment. Is that right? pan --- Barry Smith wrote: > > No, the cygwin shell and its make are used to > configure > and compile all of the PETSc libraries. > > Once the libraries are built you can switch to > Developers Studio or what you want to use to build > your > application. > > Barry > > > On Fri, 20 Jul 2007, Sumit Vaidya wrote: > > > Well I am planning to use icl and ifort compilers > on windows. > > > > Is it possible to build it without Cygwin? > > > > Sumit > > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] > > On Behalf Of Satish Balay > > Sent: Thursday, July 19, 2007 8:35 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: PETSc compiler > > > > On Thu, 19 Jul 2007, Sumit Vaidya wrote: > > > > > I am sorry. > > > > > > I want to build it on windows. I forgot to > mention that. On windows I will > > > not be having "configure.py" script. > > > > You'll have to follow the installation > instructions from: > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html > > > > you'll need to run config/configure.py [from > cygwin] on windows - > > as per the installation instructions. > > > > Sepecifically - check the instruction in > 'Additional Microsoft Windows > > Notes:' section. > > > > BTW: what compilers [c, fortran] do you plan to > use on windows? > > > > Satish > > > > > > DISCLAIMER > > ========== > > This e-mail may contain privileged and > confidential information which is the property of > Persistent Systems Pvt. Ltd. It is intended only for > the use of the individual or entity to which it is > addressed. If you are not the intended recipient, > you are not authorized to read, retain, copy, print, > distribute or use this message. If you have received > this communication in error, please notify the > sender and delete all copies of this message. > Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. > > > > > > ____________________________________________________________________________________ Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. http://farechase.yahoo.com/ From balay at mcs.anl.gov Tue Jul 24 01:35:33 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Jul 2007 01:35:33 -0500 (CDT) Subject: PETSc compiler In-Reply-To: <182703.14515.qm@web36804.mail.mud.yahoo.com> References: <182703.14515.qm@web36804.mail.mud.yahoo.com> Message-ID: On Mon, 23 Jul 2007, li pan wrote: > I also have interest about how to use petsc under > visual studio. Can you explain a little bit how to do > it? I'm using cygwin. Cygwin generates libraries for > linux environment. Is that right? Cygwin does not generate libraries for linux. Its provides a unix enviornment on windows [with shell, make, sed, python etc tools with unix paths like /usr/bin etc..] Our build tools are based on these unix tools, so we require cygwin tools to be able to compile PETSc libraries - even for use with MS compilers. [one can also use gcc/g77 etc via cygwin on windows] If you wish to use PETSc on windows with MS compilers, follow the installation instructions [It has specific instructions for windows installs] http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html Once the libraries are compiled, you can use them from [your application in] MS project files as per the instructions in 'Project Files:' section of the installation instructions. Satish From tim at cevis.uni-bremen.de Tue Jul 24 02:37:41 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Tue, 24 Jul 2007 09:37:41 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: Message-ID: Dear Lisandro, On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > On 7/23/07, Tim Kr?ger wrote: > >> > What kind of stabilization for advection and pressure are you using? >> >> Streamline diffusion (for advection). I am not aware of the >> requirement to stabilize the pressure as well. > > Well, you need to stabilize pressure if you have a non div-stable > finite element space (this happens when using linear, equal-order FE > for both velocity and pressure). Surely you are using a stable FE. I'm using Taylor-Hood tetrahedron elements. > I'm not exactly sure what do you understand for 'streamline difussion' > (is rathre general). But we normally use non div-stable FE pairs, > formulated with SUPG/PSPG (the first for advection, the second for > pressure) stabilization, as proposed by Tezduyar in many publications. I think it's the same that I am using, but I am not sure. Can you give me a helpful reference? > If you are interested in trying the my solution, I can help you. But I > cannot assert you will have good performance, specially in big 3D > problems. Anyway, we are currently using it to simulate the flow > around a racing car (Re=16e6) with about 600K nodes in 30 processors. I'm currently trying Matt's suggestion (to use MUMPS). (It seems not to easy to install MUMPS.) Anyway, I would like to try your solution as well. What would I have to do? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From gaatenek at irisa.fr Tue Jul 24 03:05:09 2007 From: gaatenek at irisa.fr (gaatenek at irisa.fr) Date: Tue, 24 Jul 2007 10:05:09 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: Message-ID: <1232.131.254.11.48.1185264309.squirrel@mail.irisa.fr> Dear Tim, I affraid that you have same memory problem while using MUMPS. I have observed same memory problem while trying to solve some 3D Stokes equations with MUMPS. I think the better way to solve this type of problem is KSP with appropriate PC. Regards, Guy > Dear Lisandro, > > On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > >> On 7/23/07, Tim Kr?ger wrote: >> >>> > What kind of stabilization for advection and pressure are you using? >>> >>> Streamline diffusion (for advection). I am not aware of the >>> requirement to stabilize the pressure as well. >> >> Well, you need to stabilize pressure if you have a non div-stable >> finite element space (this happens when using linear, equal-order FE >> for both velocity and pressure). Surely you are using a stable FE. > > I'm using Taylor-Hood tetrahedron elements. > >> I'm not exactly sure what do you understand for 'streamline difussion' >> (is rathre general). But we normally use non div-stable FE pairs, >> formulated with SUPG/PSPG (the first for advection, the second for >> pressure) stabilization, as proposed by Tezduyar in many publications. > > I think it's the same that I am using, but I am not sure. Can you > give me a helpful reference? > >> If you are interested in trying the my solution, I can help you. But I >> cannot assert you will have good performance, specially in big 3D >> problems. Anyway, we are currently using it to simulate the flow >> around a racing car (Re=16e6) with about 600K nodes in 30 processors. > > I'm currently trying Matt's suggestion (to use MUMPS). (It seems not > to easy to install MUMPS.) Anyway, I would like to try your solution > as well. What would I have to do? > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From knepley at gmail.com Tue Jul 24 04:05:10 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Jul 2007 04:05:10 -0500 Subject: Poisson Eqn in a concentric cube In-Reply-To: References: Message-ID: Currently, the DA does not directly support cutout grids, as you have shown above. However, you could just substitute the identity for all points in the cutout, and only suffer from an excessive preallocation. Does this make sense? Thanks, Matt On 7/23/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > Hi, > > I would like to solve the Poisson equation (3D) in a concentric type of > cube as in the attached figure using PETSc. The solution domain would be the > shaded region only and it extends inside the cube. I would like to have > suggestions on ways to form and assemble the matrix associated with the > discretization. Also, is it a good practise to discretize the entire domain > and blank the grid points which are within the cube not in the domain ? > > Regards, > Dominic > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From DOMI0002 at ntu.edu.sg Tue Jul 24 04:19:12 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Tue, 24 Jul 2007 17:19:12 +0800 Subject: Poisson Eqn in a concentric cube In-Reply-To: Message-ID: Thanks Matt, Yeah, I had this in mind... Since its an academic exercise, I guess its not much of a problem. Dominic -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, July 24, 2007 5:05 PM To: petsc-users at mcs.anl.gov Subject: Re: Poisson Eqn in a concentric cube Currently, the DA does not directly support cutout grids, as you have shown above. However, you could just substitute the identity for all points in the cutout, and only suffer from an excessive preallocation. Does this make sense? Thanks, Matt On 7/23/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > Hi, > > I would like to solve the Poisson equation (3D) in a concentric > type of cube as in the attached figure using PETSc. The solution > domain would be the shaded region only and it extends inside the cube. > I would like to have suggestions on ways to form and assemble the > matrix associated with the discretization. Also, is it a good practise > to discretize the entire domain and blank the grid points which are within the cube not in the domain ? > > Regards, > Dominic > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From sdettrick at gmail.com Tue Jul 24 07:07:13 2007 From: sdettrick at gmail.com (Sean Dettrick) Date: Tue, 24 Jul 2007 08:07:13 -0400 Subject: Poisson Eqn in a concentric cube Message-ID: <44114ec40707240507i3583d681k3c653f7d4f4944ae@mail.gmail.com> Just curious ... to reduce the matrix size, is it possible to build a nullspace using the cutout points, and then remove that nullspace from the matrix constructed from the DA? Or does that have some other effect? Thanks, Sean Currently, the DA does not directly support cutout grids, as you have shown above. However, you could just substitute the identity for all points in the cutout, and only suffer from an excessive preallocation. Does this make sense? Thanks, Matt On 7/23/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > Hi, > > I would like to solve the Poisson equation (3D) in a concentric type of > cube as in the attached figure using PETSc. The solution domain would be the > shaded region only and it extends inside the cube. I would like to have > suggestions on ways to form and assemble the matrix associated with the > discretization. Also, is it a good practise to discretize the entire domain > and blank the grid points which are within the cube not in the domain ? > > Regards, > Dominic > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From balay at mcs.anl.gov Tue Jul 24 10:03:44 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Jul 2007 10:03:44 -0500 (CDT) Subject: KSP/PC choice In-Reply-To: <200707240748.l6O7mfr50266@mcs.anl.gov> References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: On Tue, 24 Jul 2007, owner-petsc-users at mcs.anl.gov wrote: > Date: Tue, 24 Jul 2007 09:47:37 +0200 (CEST) > From: =?iso-8859-15?Q?Tim_Kr=F6ger?= > X-X-Sender: tim at elektrik > To: petsc-users at mcs.anl.gov > Subject: BOUNCE petsc-users at mcs.anl.gov: Message too long (>80000 chars) Its best to send installation issues involving configure.log to petsc-maint at mcs.anl.gov - and not the list. > Dear Matt, > > On Mon, 23 Jul 2007, Matthew Knepley wrote: > > > 1) Until you run out of memory, I would use sparse direct like MUMPS > > > > 2) After that, as long as you have the memory I would increase the > > GMRES vectors, say to 50 or 100. > > > > 3) After that I would try LGMRES which generally converges better on these > > problems. > > Thank you very much for your advice. I tried 2 and 3, but they did > not solve the problem. Using MUMPS fails since I am unable to > compile PETSc with MUMPS. See the attached logfile. What did I do > wrong? --with-cc=gcc --with-fc=gfortran --with-shared=0 --download-mumps=1 --with-mpi-include=/home/tim/archives/packages/mpich-1.2.7/include/ --with-mpi-lib=/home/tim/archives/packages/mpich-1.2.7/lib/libmpich.a --download-scalapack --download-blacs The MPI [lib] specification is incomplete. It gives the following errors: > /home/tim/archives/packages/petsc-2.3.3-p3/conftest.F:4: undefined reference to `mpi_init_' What do you have for 'mpicc -show' and 'mpif90 -show' If they are using gcc,gfortran, then use: ./config/configure.py --with-mpi-dir=/home/tim/archives/packages/mpich-1.2.7 --download-blacs=1 --download-scalapack=1 --download-mumps=1 [this way mpicc/mpif90 wrappers get used, and they know how to resolve mpi_init_ correctly.] Alternatively - just use the following - so that the MPI is also built with compatible compilers. --with-cc=gcc --with-fc=gfortran --download-mpich=1 Satish From dalcinl at gmail.com Tue Jul 24 13:52:04 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Jul 2007 15:52:04 -0300 Subject: KSP/PC choice In-Reply-To: References: Message-ID: > On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > I think it's the same that I am using, but I am not sure. Can you > give me a helpful reference? You can start here. This paper computes stab. params. using element mat and vecs, but you can also found explicit formulas (look for UGN subindices in formulas) http://www.mems.rice.edu/TAFSM/PUB_PRE/jALL/j89-CMAME-EBTau.pdf > I'm currently trying Matt's suggestion (to use MUMPS). (It seems not > to easy to install MUMPS.) I istall MUMPS by letting PETSc to download and install it for me. Always worked. > Anyway, I would like to try your solution as well. What would I have to do? First of all, you should move to the latest PETSc (2.3.3). Next, you have to be able to assemble a matrix of type MATIS. This implies you need to setup a ISLocalToGlobalMapping, by passing to it all local indices of degrees of freedom you have in the local processor. Once you have done this, you should be able to solve in parallel an small problem using PCJACOBI (jacobi, not block-jacobi). Next, mail me again. Using my preconditioner should then work out of the box. If you can confirm some gains, I would consider it to be added to PETSc. Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From zonexo at gmail.com Tue Jul 24 20:49:15 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 25 Jul 2007 09:49:15 +0800 Subject: KSP/PC choice In-Reply-To: References: Message-ID: <46A6AC1B.4000800@gmail.com> Hi, I am also trying to solve the NS eqns in 2d, finite volume mtd, using fractional step formulation. I am currently using KSPBCGS and PCILU to solve the momentum eqn matrix and HYPRE's AMG and the default ksptype to solve the poisson eqn. Do you ppl think this is a good choice? Or is there a better recommendation? Thank you. Lisandro Dalcin wrote: >> On Mon, 23 Jul 2007, Lisandro Dalcin wrote: >> I think it's the same that I am using, but I am not sure. Can you >> give me a helpful reference? > > You can start here. This paper computes stab. params. using element > mat and vecs, but you can also found explicit formulas (look for UGN > subindices in formulas) > > http://www.mems.rice.edu/TAFSM/PUB_PRE/jALL/j89-CMAME-EBTau.pdf > >> I'm currently trying Matt's suggestion (to use MUMPS). (It seems not >> to easy to install MUMPS.) > > I istall MUMPS by letting PETSc to download and install it for me. > Always worked. > >> Anyway, I would like to try your solution as well. What would I have >> to do? > > First of all, you should move to the latest PETSc (2.3.3). Next, you > have to be able to assemble a matrix of type MATIS. This implies you > need to setup a ISLocalToGlobalMapping, by passing to it all local > indices of degrees of freedom you have in the local processor. > Once you have done this, you should be able to solve in parallel an > small problem using PCJACOBI (jacobi, not block-jacobi). > > Next, mail me again. Using my preconditioner should then work out of > the box. If you can confirm some gains, I would consider it to be > added to PETSc. > > Regards, > > From bsmith at mcs.anl.gov Wed Jul 25 10:31:12 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 25 Jul 2007 10:31:12 -0500 (CDT) Subject: KSP/PC choice In-Reply-To: <46A6AC1B.4000800@gmail.com> References: <46A6AC1B.4000800@gmail.com> Message-ID: Ben, There is really no way to know what is the best preconditioner/Krylov approach without trying them on a representative problem. Barry On Wed, 25 Jul 2007, Ben Tay wrote: > Hi, > > I am also trying to solve the NS eqns in 2d, finite volume mtd, using > fractional step formulation. > > I am currently using KSPBCGS and PCILU to solve the momentum eqn matrix and > HYPRE's AMG and the default ksptype to solve the poisson eqn. Do you ppl think > this is a good choice? Or is there a better recommendation? > > Thank you. > > Lisandro Dalcin wrote: > > > On Mon, 23 Jul 2007, Lisandro Dalcin wrote: > > > I think it's the same that I am using, but I am not sure. Can you > > > give me a helpful reference? > > > > You can start here. This paper computes stab. params. using element > > mat and vecs, but you can also found explicit formulas (look for UGN > > subindices in formulas) > > > > http://www.mems.rice.edu/TAFSM/PUB_PRE/jALL/j89-CMAME-EBTau.pdf > > > > > I'm currently trying Matt's suggestion (to use MUMPS). (It seems not > > > to easy to install MUMPS.) > > > > I istall MUMPS by letting PETSc to download and install it for me. > > Always worked. > > > > > Anyway, I would like to try your solution as well. What would I have to > > > do? > > > > First of all, you should move to the latest PETSc (2.3.3). Next, you > > have to be able to assemble a matrix of type MATIS. This implies you > > need to setup a ISLocalToGlobalMapping, by passing to it all local > > indices of degrees of freedom you have in the local processor. > > Once you have done this, you should be able to solve in parallel an > > small problem using PCJACOBI (jacobi, not block-jacobi). > > > > Next, mail me again. Using my preconditioner should then work out of > > the box. If you can confirm some gains, I would consider it to be > > added to PETSc. > > > > Regards, > > > > > > From tim at cevis.uni-bremen.de Wed Jul 25 10:37:14 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Wed, 25 Jul 2007 17:37:14 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: Dear Satish On Tue, 24 Jul 2007, Satish Balay wrote: > Its best to send installation issues involving configure.log to > petsc-maint at mcs.anl.gov - and not the list. Okay; sorry for that. > --with-cc=gcc --with-fc=gfortran --with-shared=0 --download-mumps=1 > --with-mpi-include=/home/tim/archives/packages/mpich-1.2.7/include/ > --with-mpi-lib=/home/tim/archives/packages/mpich-1.2.7/lib/libmpich.a > --download-scalapack --download-blacs > > The MPI [lib] specification is incomplete. It gives the following errors: > >> /home/tim/archives/packages/petsc-2.3.3-p3/conftest.F:4: undefined reference to `mpi_init_' > > What do you have for 'mpicc -show' and 'mpif90 -show' mpif90 said that there was no f90 compiler configured for MPI. This obviously was the problem, so thank you for that hint. It now compiles perfectly. But now, the next problem is that I am not able to use MUMPS. As described in the manual, I start my application using -ksp_type preonly -pc_type lu -mat_type aijmumps -ksp_monitor and I get [0]PETSC ERROR: MatLUFactorSymbolic() line 1940 in src/mat/interface/matrix.c [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type mpiaij symbolic LU! [0]PETSC ERROR: PCSetUp_LU() line 242 in src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: PCSetUp() line 798 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 234 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 332 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: User provided function() line 396 in unknowndirectory/src/numerics/petsc_linear_solver.C What is wrong? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From knepley at gmail.com Wed Jul 25 10:45:54 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Jul 2007 10:45:54 -0500 Subject: KSP/PC choice In-Reply-To: References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: On 7/25/07, Tim Kr?ger wrote: > Dear Satish > > But now, the next problem is that I am not able to use MUMPS. As > described in the manual, I start my application using > > -ksp_type preonly > -pc_type lu > -mat_type aijmumps > -ksp_monitor > > and I get It appears that you never call MatSetFromOptions() on the matrix you create, which would prevent the option from changing the type. Matt > [0]PETSC ERROR: MatLUFactorSymbolic() line 1940 in src/mat/interface/matrix.c > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type mpiaij symbolic LU! > [0]PETSC ERROR: PCSetUp_LU() line 242 in src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: PCSetUp() line 798 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 234 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 332 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: User provided function() line 396 in unknowndirectory/src/numerics/petsc_linear_solver.C > > What is wrong? > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Wed Jul 25 11:20:04 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Jul 2007 13:20:04 -0300 Subject: KSP/PC choice In-Reply-To: <46A6AC1B.4000800@gmail.com> References: <46A6AC1B.4000800@gmail.com> Message-ID: On 7/24/07, Ben Tay wrote: > I am also trying to solve the NS eqns in 2d, finite volume mtd, using > fractional step formulation. > > I am currently using KSPBCGS and PCILU to solve the momentum eqn matrix > and HYPRE's AMG and the default ksptype to solve the poisson eqn. Do you > ppl think this is a good choice? Or is there a better recommendation? > This seems a godd starting point, but I would try to make sure using KSPCG for the poisson problem (of course, this if your poisson matrix is SPD, as it usually is in a fractional step methods). Additionally, if your poisson matrix is exactly the same for every time step, I believe HYPRE AMG is the best option for PC if your problem is rather large. Of course, as Barry said, all this is strongly problem dependent. You should try for the best combination. It's easy, just use command line options (of course, you have to set a different 'prefix' for your velocity and pressure solvers). Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From tim at cevis.uni-bremen.de Fri Jul 27 02:21:48 2007 From: tim at cevis.uni-bremen.de (=?iso-8859-15?Q?Tim_Kr=F6ger?=) Date: Fri, 27 Jul 2007 09:21:48 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: Dear Matt On Wed, 25 Jul 2007, Matthew Knepley wrote: >> -ksp_type preonly >> -pc_type lu >> -mat_type aijmumps >> -ksp_monitor >> > It appears that you never call MatSetFromOptions() on the matrix you > create, which would prevent the option from changing the type. Thank you very much for your help. Now, the MUMPS solver is working for me, and it really seems to solve the problem that I was working on for nearly a year now! However, I have one more question: I usually call KSPGetIterationNumber() after the solve, and if it tells me that 0 KSP iterations have been used, I assume that the initial guess was already accurate enough. In this case, I don't have to solve the system in the next time steps anymore until some change of the setting takes place (which happens from time to time in my application). This saves a lot of computational time. However, with the MUMPS solver, KSPGetIterationNumber() never supplies 0. Is there any possibility to find out whether the initial guess was accurate enough? (It might be noteworthy that I use "-ksptype richardson" instead of "-ksptype preonly" since in the latter case I get an error message due to my non-zero initial guess.) Also, I would like to know whether there is any possibility to monitor the convergence of the MUMPS solver (analogously to -kspmonitor). Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From gaatenek at irisa.fr Fri Jul 27 03:17:02 2007 From: gaatenek at irisa.fr (gaatenek at irisa.fr) Date: Fri, 27 Jul 2007 10:17:02 +0200 (CEST) Subject: KSP/PC choice In-Reply-To: References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: <3901.131.254.11.48.1185524222.squirrel@mail.irisa.fr> Dear Tims, MUMPS is direct solver so you do not need Krylov subspace which are generally iterative solver. May be you are using MUMPS as solver in you problem not as preconditionner. Regards > Dear Matt > > On Wed, 25 Jul 2007, Matthew Knepley wrote: > >>> -ksp_type preonly >>> -pc_type lu >>> -mat_type aijmumps >>> -ksp_monitor >>> >> It appears that you never call MatSetFromOptions() on the matrix you >> create, which would prevent the option from changing the type. > > Thank you very much for your help. Now, the MUMPS solver is working > for me, and it really seems to solve the problem that I was working on > for nearly a year now! > > However, I have one more question: I usually call > KSPGetIterationNumber() after the solve, and if it tells me that 0 KSP > iterations have been used, I assume that the initial guess was already > accurate enough. In this case, I don't have to solve the system in > the next time steps anymore until some change of the setting takes > place (which happens from time to time in my application). This saves > a lot of computational time. However, with the MUMPS solver, > KSPGetIterationNumber() never supplies 0. Is there any possibility to > find out whether the initial guess was accurate enough? > > (It might be noteworthy that I use "-ksptype richardson" instead of > "-ksptype preonly" since in the latter case I get an error message due > to my non-zero initial guess.) > > Also, I would like to know whether there is any possibility to monitor > the convergence of the MUMPS solver (analogously to -kspmonitor). > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim at mevis.de, tim at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, D-28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > From gkells at thphys.nuim.ie Fri Jul 27 06:19:33 2007 From: gkells at thphys.nuim.ie (Graham Kells) Date: Fri, 27 Jul 2007 12:19:33 +0100 Subject: Scattering context options Message-ID: <46A9D4C5.6020001@thphys.nuim.ie> Hi , I'm scattering from a global vector (used as a kind of look up table) to a bunch of local vectors and then using the values in the local vectors as indices to populate a global matrix. The basic population routine goes something like this do myrow=Istart,Iend-1 ! Returns column indices of full matrix for rowindex list(myrow+1) call getrow(list(myrow+1),rowCOO,colCOO,valCOO) call ISCreateGeneral(PETSC_COMM_WORLD,n,(colCOO-1),from,ierr) call ISCreateGeneral(PETSC_COMM_SELF,n,idx_to-1,to,ierr) call VecScatterCreate(vlist,from,outvec,to,scatter,ierr) call VecScatterBegin(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterEnd(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) call VecScatterDestroy(scatter,ierr) call ISDestroy(from,ierr) call ISDestroy(to,ierr) call VecGetValues(outvec,n,idx_to-1,idx_real,ierr) idx_from=int(idx_real) call MatSetValues(A,1,myrow,n,idx_from-1,valCOO,INSERT_VALUES,ierr) end do While this works, it is prohibitively slow. Any ideas on why this is? Of course if you can suggest a better way of doing this that would be great. Supposing you can't and I want to experiment with different MPI communication modes within the scatter context. How do specify the MPI_Ssend option, for example? Thanks in advance, Graham From stephan.kramer at imperial.ac.uk Fri Jul 27 05:58:14 2007 From: stephan.kramer at imperial.ac.uk (Stephan Kramer) Date: Fri, 27 Jul 2007 11:58:14 +0100 Subject: KSP/PC choice In-Reply-To: References: <200707240748.l6O7mfr50266@mcs.anl.gov> Message-ID: <1185533894.4083.6.camel@ese-luigi.ese.ic.ac.uk> Hi Tim, As Guy said MUMPS is a direct solver, i.e. it constructs the inverse of the matrix (in some sense) and applies that to the rhs. This means first of all you need quite a lot of memory, as you need to store a dense matrix, but also that there is no iterative process. You do not/should not supply an initial guess. ksp_type should really be preonly, anything else doesn't make sense. As there are no iterations, it also doesn't return a number of iterations and there is also no convergence to monitor, Hope this helps Cheers Stephan On Fri, 2007-07-27 at 09:21 +0200, Tim Kr?ger wrote: > Dear Matt > > On Wed, 25 Jul 2007, Matthew Knepley wrote: > > >> -ksp_type preonly > >> -pc_type lu > >> -mat_type aijmumps > >> -ksp_monitor > >> > > It appears that you never call MatSetFromOptions() on the matrix you > > create, which would prevent the option from changing the type. > > Thank you very much for your help. Now, the MUMPS solver is working > for me, and it really seems to solve the problem that I was working on > for nearly a year now! > > However, I have one more question: I usually call > KSPGetIterationNumber() after the solve, and if it tells me that 0 KSP > iterations have been used, I assume that the initial guess was already > accurate enough. In this case, I don't have to solve the system in > the next time steps anymore until some change of the setting takes > place (which happens from time to time in my application). This saves > a lot of computational time. However, with the MUMPS solver, > KSPGetIterationNumber() never supplies 0. Is there any possibility to > find out whether the initial guess was accurate enough? > > (It might be noteworthy that I use "-ksptype richardson" instead of > "-ksptype preonly" since in the latter case I get an error message due > to my non-zero initial guess.) > > Also, I would like to know whether there is any possibility to monitor > the convergence of the MUMPS solver (analogously to -kspmonitor). > > Best Regards, > > Tim > From knepley at gmail.com Fri Jul 27 08:00:40 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 27 Jul 2007 08:00:40 -0500 Subject: KSP/PC choice In-Reply-To: <1185533894.4083.6.camel@ese-luigi.ese.ic.ac.uk> References: <200707240748.l6O7mfr50266@mcs.anl.gov> <1185533894.4083.6.camel@ese-luigi.ese.ic.ac.uk> Message-ID: You could, of course, embed MUMPS as a preconditioner in a Krylov iteration like GMRES and getthe effect that you want. Mat On 7/27/07, Stephan Kramer wrote: > Hi Tim, > > As Guy said MUMPS is a direct solver, i.e. it constructs the inverse of > the matrix (in some sense) and applies that to the rhs. This means first > of all you need quite a lot of memory, as you need to store a dense > matrix, but also that there is no iterative process. You do not/should > not supply an initial guess. ksp_type should really be preonly, anything > else doesn't make sense. As there are no iterations, it also doesn't > return a number of iterations and there is also no convergence to > monitor, > > Hope this helps > > Cheers > Stephan > > On Fri, 2007-07-27 at 09:21 +0200, Tim Kr?ger wrote: > > Dear Matt > > > > On Wed, 25 Jul 2007, Matthew Knepley wrote: > > > > >> -ksp_type preonly > > >> -pc_type lu > > >> -mat_type aijmumps > > >> -ksp_monitor > > >> > > > It appears that you never call MatSetFromOptions() on the matrix you > > > create, which would prevent the option from changing the type. > > > > Thank you very much for your help. Now, the MUMPS solver is working > > for me, and it really seems to solve the problem that I was working on > > for nearly a year now! > > > > However, I have one more question: I usually call > > KSPGetIterationNumber() after the solve, and if it tells me that 0 KSP > > iterations have been used, I assume that the initial guess was already > > accurate enough. In this case, I don't have to solve the system in > > the next time steps anymore until some change of the setting takes > > place (which happens from time to time in my application). This saves > > a lot of computational time. However, with the MUMPS solver, > > KSPGetIterationNumber() never supplies 0. Is there any possibility to > > find out whether the initial guess was accurate enough? > > > > (It might be noteworthy that I use "-ksptype richardson" instead of > > "-ksptype preonly" since in the latter case I get an error message due > > to my non-zero initial guess.) > > > > Also, I would like to know whether there is any possibility to monitor > > the convergence of the MUMPS solver (analogously to -kspmonitor). > > > > Best Regards, > > > > Tim > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From knepley at gmail.com Fri Jul 27 08:11:21 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 27 Jul 2007 08:11:21 -0500 Subject: Scattering context options In-Reply-To: <46A9D4C5.6020001@thphys.nuim.ie> References: <46A9D4C5.6020001@thphys.nuim.ie> Message-ID: Maybe you can give more high level motivation for this setup. Without knowing anything, it appears that you are communicating almost every entry of your matrix, sometimes more than once. This will definitely be very very slow. Parallel computing (at least scalable computing) relies on a model where most work and memory access is local. Its not clear to me that you have this. If not, parallel computing will not help very much. However, a reorganization might help considerably. For one, fewer, larger communications is always better. Matt On 7/27/07, Graham Kells wrote: > Hi , > > I'm scattering from a global vector (used as a kind of look up table) > to a bunch of local vectors and then using the values in the local > vectors as indices to populate a global matrix. > > The basic population routine goes something like this > > do myrow=Istart,Iend-1 > > > ! Returns column indices of full matrix for rowindex list(myrow+1) > call getrow(list(myrow+1),rowCOO,colCOO,valCOO) > > > call ISCreateGeneral(PETSC_COMM_WORLD,n,(colCOO-1),from,ierr) > call ISCreateGeneral(PETSC_COMM_SELF,n,idx_to-1,to,ierr) > call VecScatterCreate(vlist,from,outvec,to,scatter,ierr) > call > VecScatterBegin(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) > call > VecScatterEnd(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) > call VecScatterDestroy(scatter,ierr) > call ISDestroy(from,ierr) > call ISDestroy(to,ierr) > > call VecGetValues(outvec,n,idx_to-1,idx_real,ierr) > idx_from=int(idx_real) > > > call MatSetValues(A,1,myrow,n,idx_from-1,valCOO,INSERT_VALUES,ierr) > > end do > > While this works, it is prohibitively slow. Any ideas on why this is? > Of course if you can suggest a better way of doing this that would be great. > > Supposing you can't and I want to experiment with different MPI > communication modes within the scatter context. How do specify the > MPI_Ssend option, for example? > > Thanks in advance, > > Graham > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Fri Jul 27 12:17:34 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Jul 2007 12:17:34 -0500 (CDT) Subject: Scattering context options In-Reply-To: <46A9D4C5.6020001@thphys.nuim.ie> References: <46A9D4C5.6020001@thphys.nuim.ie> Message-ID: Graham, It seems unlike the matrix was preallocated; this will definitely slow things down a great deal. Pluse you have a scatter for each row, this is fundamentally not scalable. Barry On Fri, 27 Jul 2007, Graham Kells wrote: > Hi , > > I'm scattering from a global vector (used as a kind of look up table) to a > bunch of local vectors and then using the values in the local vectors as > indices to populate a global matrix. > The basic population routine goes something like this > > do myrow=Istart,Iend-1 > > ! Returns column indices of full matrix for rowindex list(myrow+1) > call getrow(list(myrow+1),rowCOO,colCOO,valCOO) > call ISCreateGeneral(PETSC_COMM_WORLD,n,(colCOO-1),from,ierr) > call ISCreateGeneral(PETSC_COMM_SELF,n,idx_to-1,to,ierr) call > VecScatterCreate(vlist,from,outvec,to,scatter,ierr) > call > VecScatterBegin(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) > call VecScatterEnd(scatter,vlist,outvec,INSERT_VALUES,SCATTER_FORWARD,ierr) > call VecScatterDestroy(scatter,ierr) > call ISDestroy(from,ierr) > call ISDestroy(to,ierr) > > call VecGetValues(outvec,n,idx_to-1,idx_real,ierr) > idx_from=int(idx_real) > > > call MatSetValues(A,1,myrow,n,idx_from-1,valCOO,INSERT_VALUES,ierr) > > end do > > While this works, it is prohibitively slow. Any ideas on why this is? Of > course if you can suggest a better way of doing this that would be great. > > Supposing you can't and I want to experiment with different MPI communication > modes within the scatter context. How do specify the MPI_Ssend option, for > example? > > Thanks in advance, > > Graham > > > > > > > > > > > > From zonexo at gmail.com Mon Jul 30 23:48:44 2007 From: zonexo at gmail.com (Ben Tay) Date: Tue, 31 Jul 2007 12:48:44 +0800 Subject: How is the sys of linear eqns solved using PETSc in this application? Message-ID: <804ab5d40707302148x7014ea26x3e5bfa73bf6d6ff@mail.gmail.com> Hi, I'm formulating a NS solver which computes the flow past an airfoil. The airfoil has a c-grid and it overlaps with the background cartesian grid. In other words, it is a chimera or overset grid application. The boundary cells of the c-grid are connected to the cartesian grid thru interpolation stencil. One way to solve the problem is to solved implicitly ie for each cell phi(c-grid)=a1*phi(cart,1)+a2*phi(cart,2)+a3*phi(cart,3)+a4*phi(cart,4) and vice versa for the phi(cart). This is entered into the sys of linear eqns comprising of c-grid and cartesian and solved all at once. Another way is to solve the cartesian grid eqns fully 1st, compute phi(c-grid) using the known phi(cart,*) and solve the c-grid fully ie explicit solving. Is it possible to do this in PETSc: 1. Do 1 iteration of cartesian grid eqns 2. compute phi(c-grid) using the newly iterated phi(cart,*) values 3. Do 1 iteration of c-grid eqns 4. compute phi(cart) using the newly iterated phi(c-grid,*) values 5. go back to 1. Or is this mtd the same as solving the whole sys implicitly (ie 1st case)? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From sumit_vaidya at persistent.co.in Tue Jul 31 06:47:11 2007 From: sumit_vaidya at persistent.co.in (Sumit Vaidya) Date: Tue, 31 Jul 2007 17:17:11 +0530 Subject: "if" statement in "$PESC_DIR\bmake\common\rules" makefile Message-ID: <000001c7d368$880b7420$4598580a@persistent.co.in> Hi, I would like to understand the rules makefile. What are these various options '-d', '-o', '-f' used in "if" statement. For example in "tree" target, the first command is "@if [ "$(DIRS)" != "" -o -d "ftn-auto" -o -d "ftn-custom" -o -d "f90-custom" ];" I would like to understand the meaning of this. What is the significance of -o -d option in "if" statement. Can you point me to some link where I will get the information about this? Waiting for your reply, Sumit DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Pvt. Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Pvt. Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 31 08:41:27 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 31 Jul 2007 08:41:27 -0500 Subject: "if" statement in "$PESC_DIR\bmake\common\rules" makefile In-Reply-To: <000001c7d368$880b7420$4598580a@persistent.co.in> References: <000001c7d368$880b7420$4598580a@persistent.co.in> Message-ID: -o means logical or and -d means does the directory exist. I believe all this is covered in the bash manpage. Matt On 7/31/07, Sumit Vaidya wrote: > > > > > Hi, > > > > I would like to understand the rules makefile. What are these various > options '-d', '-o', '-f' used in "if" statement. > > For example in "tree" target, the first command is > > "@if [ "$(DIRS)" != "" -o -d "ftn-auto" -o -d "ftn-custom" -o -d > "f90-custom" ];" > > > > I would like to understand the meaning of this. What is the significance of > ?o ?d option in "if" statement. > > Can you point me to some link where I will get the information about this? > > > > Waiting for your reply, > > Sumit > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Pvt. Ltd. It is > intended only for the use of the individual or entity to which it is > addressed. If you are not the intended recipient, you are not authorized to > read, retain, copy, print, distribute or use this message. If you have > received this communication in error, please notify the sender and delete > all copies of this message. Persistent Systems Pvt. Ltd. does not accept any > liability for virus infected mails. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From knepley at gmail.com Tue Jul 31 08:45:32 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 31 Jul 2007 08:45:32 -0500 Subject: How is the sys of linear eqns solved using PETSc in this application? In-Reply-To: <804ab5d40707302148x7014ea26x3e5bfa73bf6d6ff@mail.gmail.com> References: <804ab5d40707302148x7014ea26x3e5bfa73bf6d6ff@mail.gmail.com> Message-ID: PETSc only solves algebraic equations, although the DA construct can handle Cartesian meshes. Thus it will solve either of these systems if you set it up. Explicit methods would probably use TS, and implicit DMMG. Matt On 7/30/07, Ben Tay wrote: > Hi, > > I'm formulating a NS solver which computes the flow past an airfoil. The > airfoil has a c-grid and it overlaps with the background cartesian grid. In > other words, it is a chimera or overset grid application. > > The boundary cells of the c-grid are connected to the cartesian grid thru > interpolation stencil. One way to solve the problem is to solved implicitly > ie for each cell > phi(c-grid)=a1*phi(cart,1)+a2*phi(cart,2)+a3*phi(cart,3)+a4*phi(cart,4) > and vice versa for the phi(cart). This is entered into the sys of linear > eqns comprising of c-grid and cartesian and solved all at once. > > Another way is to solve the cartesian grid eqns fully 1st, compute > phi(c-grid) using the known phi(cart,*) and solve the c-grid fully ie > explicit solving. > > Is it possible to do this in PETSc: > > 1. Do 1 iteration of cartesian grid eqns > 2. compute phi(c-grid) using the newly iterated phi(cart,*) values > 3. Do 1 iteration of c-grid eqns > 4. compute phi(cart) using the newly iterated phi(c-grid,*) values > 5. go back to 1. > > Or is this mtd the same as solving the whole sys implicitly (ie 1st case)? > > Thanks -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener