From barryfsmith at me.com  Wed Jan  1 14:10:11 2014
From: barryfsmith at me.com (Barry Smith)
Date: Wed, 01 Jan 2014 14:10:11 -0600
Subject: [petsc-dev] get rid of petscsys.hh?
In-Reply-To: <CAMYG4Gm_cCbJ5rgOQA7AD0_8JbkaDeiAd9STHZ7JSZ0pu_4tKQ@mail.gmail.com>
References: <B5D75DB4-1CBC-4C81-B803-A9FEE12254F6@me.com>
	<CAMYG4Gm_cCbJ5rgOQA7AD0_8JbkaDeiAd9STHZ7JSZ0pu_4tKQ@mail.gmail.com>
Message-ID: <767392FC-C62E-46B3-810A-147AF9F3BB0B@me.com>


  Matt,

   You removed petscsys.hh but also need to remove its dependencies 

        CXX arch-pylith-gcc-opt/obj/src/sys/error/err.o
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c: In function 'void PetscCxxErrorThrow()':
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:304:5: error: 'ostringstream' is not a member of 'std'
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc
/petsc-pylith/binaries/src/sys/error/err.c:304:25: error: 'msg' was not declared in this scope
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:12: error: 'ostringstream' is not a member of 'std'

/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:31: error: expected primary-expression before ')' token
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:33: error: expected ';' before 'eh'
/Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:309:9: error: 'PETSc' has not been declared
make[2]: *** [arch-pylith-gcc-opt/obj/src/sys/error/err.o] Error 1

On Dec 26, 2013, at 2:59 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Dec 26, 2013 at 2:41 PM, Barry Smith <barryfsmith at me.com> wrote:
> 
> 
>     Can we get rid of petscsys.hh which still? seems to be lurking in master?
> 
> I will get rid of it.
> 
>   Matt
>  
>    Thanks
> 
>    Barry
> 
> /* Special support for C++ */
> #if defined(PETSC_CLANGUAGE_CXX) && defined(__cplusplus)
> #include <petscsys.hh>
> #endif
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From bsmith at mcs.anl.gov  Wed Jan  1 15:09:01 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 1 Jan 2014 15:09:01 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
Message-ID: <7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>


  Satish,

    Resending this. Should be really straightforward, please let me know if there are difficulties in getting this done soon.

    Thanks

     Barry

Request-assigned: balay add Report Typos and Errors link to all manual pages

On Dec 10, 2013, at 11:11 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   Satish,
> 
>    Please add to the generation of all manual pages (in the upper right corner) a Report Typos and Errors link using mailto:  For example something like
> 
> <a href=?mailto:petsc-maint at mcs.anl.gov?subject=Typo or Error in MANUALPAGE &body=Please describe the typo or error in the manual page">Report Typos and Errors</a>
> 
> where MANUALPAGE is the name of that generated manual page.
> 
>  Thanks
> 
>   Barry
> 



From jedbrown at mcs.anl.gov  Thu Jan  2 10:03:20 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 02 Jan 2014 10:03:20 -0600
Subject: [petsc-dev] IMP prototype
In-Reply-To: <58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
Message-ID: <878uuyweuv.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:

> On Dec 31, 2013, at 3:10 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Hmm, I don't really see the API for specifying the distributions.
>> rightshift->operate(">>1") is a pretty special-purpose interface.
>> Presumably this part is non-collective,
>
> No, it's collective. As I argue in section 1.1, I want sequential
> semantics.

Okay, but surely a more general interface for describing the
distribution must involve setting non-collectively (at least the data is
not collective, so you'll have to build the two-sided representation).

> It's actually not all that special purpose. The shift notation is
> enough if you have stencils with offsets that are constant during
> program execution.

Uh, only for structured grids, and for multiple dimensions, you either
need this communication abstraction to know about multi-dimensional
layout (essentially putting DMDA inside VecScatter) or you can't use the
shift notation.

> And you did read section 2.3, right?

Yeah, it's cute but limited and IMO, not very useful in practice.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/8421b836/attachment.sig>

From knepley at gmail.com  Thu Jan  2 10:10:22 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 2 Jan 2014 10:10:22 -0600
Subject: [petsc-dev] get rid of petscsys.hh?
In-Reply-To: <767392FC-C62E-46B3-810A-147AF9F3BB0B@me.com>
References: <B5D75DB4-1CBC-4C81-B803-A9FEE12254F6@me.com>
	<CAMYG4Gm_cCbJ5rgOQA7AD0_8JbkaDeiAd9STHZ7JSZ0pu_4tKQ@mail.gmail.com>
	<767392FC-C62E-46B3-810A-147AF9F3BB0B@me.com>
Message-ID: <CAMYG4Gnw8vntbbcWZQPzSYS2bjEfbq4Ta2Y1nzqS3K4vZ_xxcg@mail.gmail.com>

On Wed, Jan 1, 2014 at 2:10 PM, Barry Smith <barryfsmith at me.com> wrote:

>
>   Matt,
>
>    You removed petscsys.hh but also need to remove its dependencies
>

I don't think we can remove this. I fixed the include.

   Matt


>         CXX arch-pylith-gcc-opt/obj/src/sys/error/err.o
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:
> In function 'void PetscCxxErrorThrow()':
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:304:5:
> error: 'ostringstream' is not a member of 'std'
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc
> /petsc-pylith/binaries/src/sys/error/err.c:304:25: error: 'msg' was not
> declared in this scope
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:12:
> error: 'ostringstream' is not a member of 'std'
>
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:31:
> error: expected primary-expression before ')' token
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:33:
> error: expected ';' before 'eh'
> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:309:9:
> error: 'PETSc' has not been declared
> make[2]: *** [arch-pylith-gcc-opt/obj/src/sys/error/err.o] Error 1
>
> On Dec 26, 2013, at 2:59 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> > On Thu, Dec 26, 2013 at 2:41 PM, Barry Smith <barryfsmith at me.com> wrote:
> >
> >
> >     Can we get rid of petscsys.hh which still? seems to be lurking in
> master?
> >
> > I will get rid of it.
> >
> >   Matt
> >
> >    Thanks
> >
> >    Barry
> >
> > /* Special support for C++ */
> > #if defined(PETSC_CLANGUAGE_CXX) && defined(__cplusplus)
> > #include <petscsys.hh>
> > #endif
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/02bbf728/attachment.html>

From eijkhout at tacc.utexas.edu  Thu Jan  2 10:31:57 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Thu, 2 Jan 2014 16:31:57 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <878uuyweuv.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
Message-ID: <DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>


On Jan 2, 2014, at 10:03 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> only for

Look, it's a demonstration prototype. The shift syntax handles one case. For other cases you need a different syntax or extend the current.

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From barryfsmith at me.com  Thu Jan  2 10:37:56 2014
From: barryfsmith at me.com (Barry Smith)
Date: Thu, 02 Jan 2014 10:37:56 -0600
Subject: [petsc-dev] get rid of petscsys.hh?
In-Reply-To: <CAMYG4Gnw8vntbbcWZQPzSYS2bjEfbq4Ta2Y1nzqS3K4vZ_xxcg@mail.gmail.com>
References: <B5D75DB4-1CBC-4C81-B803-A9FEE12254F6@me.com>
	<CAMYG4Gm_cCbJ5rgOQA7AD0_8JbkaDeiAd9STHZ7JSZ0pu_4tKQ@mail.gmail.com>
	<767392FC-C62E-46B3-810A-147AF9F3BB0B@me.com>
	<CAMYG4Gnw8vntbbcWZQPzSYS2bjEfbq4Ta2Y1nzqS3K4vZ_xxcg@mail.gmail.com>
Message-ID: <58CD0402-08CE-4650-B831-BB1CFD6C9FB8@me.com>

How about just embedding the information directly into the petscsys.h file and putting a comment explaining its purpose


> On Jan 2, 2014, at 10:10 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> 
>> On Wed, Jan 1, 2014 at 2:10 PM, Barry Smith <barryfsmith at me.com> wrote:
>> 
>>   Matt,
>> 
>>    You removed petscsys.hh but also need to remove its dependencies
> 
> I don't think we can remove this. I fixed the include.
> 
>    Matt
>  
>>         CXX arch-pylith-gcc-opt/obj/src/sys/error/err.o
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c: In function 'void PetscCxxErrorThrow()':
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:304:5: error: 'ostringstream' is not a member of 'std'
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc
>> /petsc-pylith/binaries/src/sys/error/err.c:304:25: error: 'msg' was not declared in this scope
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:12: error: 'ostringstream' is not a member of 'std'
>> 
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:31: error: expected primary-expression before ')' token
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:305:33: error: expected ';' before 'eh'
>> /Users/buildbot/slave/build/pylith_darwin_10.6_binbot/PETSc/petsc-pylith/binaries/src/sys/error/err.c:309:9: error: 'PETSc' has not been declared
>> make[2]: *** [arch-pylith-gcc-opt/obj/src/sys/error/err.o] Error 1
>> 
>> On Dec 26, 2013, at 2:59 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> > On Thu, Dec 26, 2013 at 2:41 PM, Barry Smith <barryfsmith at me.com> wrote:
>> >
>> >
>> >     Can we get rid of petscsys.hh which still? seems to be lurking in master?
>> >
>> > I will get rid of it.
>> >
>> >   Matt
>> >
>> >    Thanks
>> >
>> >    Barry
>> >
>> > /* Special support for C++ */
>> > #if defined(PETSC_CLANGUAGE_CXX) && defined(__cplusplus)
>> > #include <petscsys.hh>
>> > #endif
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> > -- Norbert Wiener
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/fb6db4f1/attachment.html>

From jedbrown at mcs.anl.gov  Thu Jan  2 10:50:46 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 02 Jan 2014 10:50:46 -0600
Subject: [petsc-dev] IMP prototype
In-Reply-To: <DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
Message-ID: <8738l6wcnt.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:

> On Jan 2, 2014, at 10:03 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> only for
>
> Look, it's a demonstration prototype. The shift syntax handles one
> case. For other cases you need a different syntax or extend the
> current.

Then the proposal is for something else.  The distinguishing feature of
successful programming abstractions is never one of making easy things
pretty.  There are many ways to do that, just as there are many ways to
solve a homogeneous Laplacian.  A successful abstraction needs to make
hard things possible.  As stated earlier, my principle reservation with
IMP is that adding another layer of callbacks has a long track record of
being more difficult to reason about, and I'm far from convinced that it
makes the model more expressive.  The syntax for describing
distributions is a distraction (syntax usually is), but the semantics
matter.  Two-sided descriptions are a usability problem, but a
performance benefit.  If IMP turns out not to be suitable for a
particular use case, transforming from callback-driven to non-callback
is a nontrivial refactoring, as is going the other direction.  That's
not good because it increases the overhead of incremental adoption.

I find simple demonstrations as unconvincing as most patents.  99% of
the work remains in extending the idea into something practical.  It may
or may not pan out, but we can't say anything from the simple
demonstration alone.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/0f2ebd82/attachment.sig>

From jedbrown at mcs.anl.gov  Thu Jan  2 10:52:25 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 02 Jan 2014 10:52:25 -0600
Subject: [petsc-dev] get rid of petscsys.hh?
In-Reply-To: <58CD0402-08CE-4650-B831-BB1CFD6C9FB8@me.com>
References: <B5D75DB4-1CBC-4C81-B803-A9FEE12254F6@me.com>
	<CAMYG4Gm_cCbJ5rgOQA7AD0_8JbkaDeiAd9STHZ7JSZ0pu_4tKQ@mail.gmail.com>
	<767392FC-C62E-46B3-810A-147AF9F3BB0B@me.com>
	<CAMYG4Gnw8vntbbcWZQPzSYS2bjEfbq4Ta2Y1nzqS3K4vZ_xxcg@mail.gmail.com>
	<58CD0402-08CE-4650-B831-BB1CFD6C9FB8@me.com>
Message-ID: <87zjneuy0m.fsf@jedbrown.org>

Barry Smith <barryfsmith at me.com> writes:

> How about just embedding the information directly into the petscsys.h
> file and putting a comment explaining its purpose

I'd much rather have err.c include <sstream> than petscsys.h.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/54d76691/attachment.sig>

From eijkhout at tacc.utexas.edu  Thu Jan  2 11:09:36 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Thu, 2 Jan 2014 17:09:36 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <8738l6wcnt.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
Message-ID: <E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>


On Jan 2, 2014, at 10:50 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> I find simple demonstrations as unconvincing as most patents.  99% of
> the work remains in extending the idea into something practical.  It may
> or may not pan out, but we can't say anything from the simple
> demonstration alone.

Maybe you and I disagree on what I'm demonstrating. My goal was to show that my notion of parallelism generalizes MPI & tasking notions. Not that I have a better notation for VecScatters.

And from this demonstration we can definitely say something: namely that I've shown how one API can address multiple types of parallelism. That's more than any other system I know of. The most interesting question to me is if I can do heterogeneity. There I have to wave my hands somewhat vigorously.

But let's be constructive: I want to use this demonstration to get funding. NSF/DOE/Darpa, I don't know. Now if you can't say anything from this simple demonstration, then what would convince you as a reviewer?

> another layer of callbacks

If you have mentioned that objection before it escaped my attention. Yes, I agree that in that respect (which has little to do with the parallelism part) my demonstration is not optimal. The unification of MPI & tasks is going too far there. For MPI it would be possible to have calls like VecScatterBegin/End and instead of a callback just have the local node code in place. For task models that is not possible (afaik). See for instance Quark, where each task contains a function pointer and a few data pointers.

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From jedbrown at mcs.anl.gov  Thu Jan  2 22:12:57 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 02 Jan 2014 22:12:57 -0600
Subject: [petsc-dev] IMP prototype
In-Reply-To: <E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
Message-ID: <871u0pvh2u.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:

> On Jan 2, 2014, at 10:50 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> I find simple demonstrations as unconvincing as most patents.  99% of
>> the work remains in extending the idea into something practical.  It may
>> or may not pan out, but we can't say anything from the simple
>> demonstration alone.
>
> Maybe you and I disagree on what I'm demonstrating. My goal was to
> show that my notion of parallelism generalizes MPI & tasking
> notions. Not that I have a better notation for VecScatters.

My impression is that your transformation is recognizing a common
pattern of communication into temporary buffers, followed by
computation, followed by post-communication and putting a declarative
syntax on it (with callbacks for the computation).  The same operations
can also be written imperatively and I'm not seeing the profound
advantage of converting to your callback system.


> And from this demonstration we can definitely say something: namely
> that I've shown how one API can address multiple types of
> parallelism. That's more than any other system I know of. 

There are systems like AMPI that run MPI programs on top of threads, and
the MPI implementations already optimize shared-memory communication.
If you want separate work buffers per thread, those systems will give
you a single abstraction.  But the reason people want native interfaces
to threads is so that they can use large shared data structures and
techniques like cooperative prefetch.  Your abstraction is not uniform
if you need to index into owned parts of shared data structures or
perform optimizations like cooperative prefetch.  If you're going to use
separate buffers, why is a system like MPI not sufficient?

What semantic does your abstraction provide for hybrid
distributed/shared memory that imperative communication systems cannot?

> But let's be constructive: I want to use this demonstration to get
> funding. NSF/DOE/Darpa, I don't know. Now if you can't say anything
> from this simple demonstration, then what would convince you as a
> reviewer?

Make a precise and concrete (falsifiable) statement about what semantic
your system can provide that others cannot.  Show examples that are hard
to express with existing solutions (such as MPI), but are cleanly
represented by your system.  Be fair and show examples of the converse
if they exist (silver bullets are rare).

>> another layer of callbacks
>
> If you have mentioned that objection before it escaped my attention. 

See earlier message on "Callback Hell".

> Yes, I agree that in that respect (which has little to do with the
> parallelism part) my demonstration is not optimal. The unification of
> MPI & tasks is going too far there. For MPI it would be possible to
> have calls like VecScatterBegin/End and instead of a callback just
> have the local node code in place. For task models that is not
> possible (afaik). See for instance Quark, where each task contains a
> function pointer and a few data pointers.

Quark does this because it is a dynamic scheduler.  (When people
compare, static schedules are usually as good or better, though if the
dependency graph changes from run to run, you may still want to write it
as a DAG and have callbacks.)  But your model is not a general DAG and
the execution model is BSP so it's not clear what is gained by switching
to callbacks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140102/5a08bff7/attachment.sig>

From eijkhout at tacc.utexas.edu  Thu Jan  2 23:36:06 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Fri, 3 Jan 2014 05:36:06 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <871u0pvh2u.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
Message-ID: <A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>


On Jan 2, 2014, at 10:12 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> the execution model is BSP

No it's not. There are no barriers or syncs.

> it's not clear what is gained by switching
> to callbacks.

The callbacks are there just to make the code uniform. I've edited the document to reflect that in the MPI case you can dispense with them.

> your transformation is recognizing a common
> pattern of communication into temporary buffers, followed by
> computation, followed by post-communication and putting a declarative
> syntax on it

Somewhat simplified, but not wrong. I'm kind of interested in the question what practically relevant algorithms do not conform to that model.

> (with callbacks for the computation)

They are incidental, at least in the MPI case. Ignore them.

> Your abstraction is not uniform
> if you need to index into owned parts of shared data structures or
> perform optimizations like cooperative prefetch.

Not sure what you're saying here. In case you dug into my code deeply, at the moment gathering the halo region involves one send-to-self that should be optimized away in the future. Look, it's a prototype, all right?

>  If you're going to use
> separate buffers, why is a system like MPI not sufficient?

1. No idea why you're so fixated on buffers. I've just been going for the simplest implementation that makes it work on this one example. It can all be optimized.
2. Why is MPI not sufficient: because you have to spell out too much. Calculating a halo in the case of a block-cyclically distributed vector is way too much work. An extension of IMP would make this very easy to specify, and all the hard work is done under the covers.

> What semantic does your abstraction provide for hybrid
> distributed/shared memory that imperative communication systems cannot?

How about having the exact same code that I've shown you, except that you specify that the processes are organized as N nodes times C cores? Right now I've not implemented hybrid programming, but it shouldn't be hard.

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From jedbrown at mcs.anl.gov  Fri Jan  3 07:16:39 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 03 Jan 2014 07:16:39 -0600
Subject: [petsc-dev] IMP prototype
In-Reply-To: <A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
Message-ID: <87wqihtdc8.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:

> On Jan 2, 2014, at 10:12 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> the execution model is BSP
>
> No it's not. There are no barriers or syncs.

You have a communication phase followed by computation, followed by more
communication (in general).  Looks like BSP without explicit barriers
(which would be semantically meaningless if you added them).  An example
of a less structured communication pattern is Mark's asynchronous
Gauss-Seidel.

  http://crd.lbl.gov/assets/Uploads/ANAG/gs.pdf

> The callbacks are there just to make the code uniform. I've edited the
> document to reflect that in the MPI case you can dispense with them.

I thought you were targeting hybrid MPI/threads?

>> your transformation is recognizing a common
>> pattern of communication into temporary buffers, followed by
>> computation, followed by post-communication and putting a declarative
>> syntax on it
>
> Somewhat simplified, but not wrong. I'm kind of interested in the
> question what practically relevant algorithms do not conform to that
> model.

The GS above is one example.  More simply, how does MatMult_MPI look in
your model (note overlapped communication and computation)?  Also, you
can't implement an efficient setup for your communication pattern
without richer semantics, see PetscCommBuildTwoSided_Ibarrier and the paper

  http://unixer.de/publications/img/hoefler-dsde-protocols.pdf

>> Your abstraction is not uniform
>> if you need to index into owned parts of shared data structures or
>> perform optimizations like cooperative prefetch.
>
> Not sure what you're saying here. In case you dug into my code deeply,
> at the moment gathering the halo region involves one send-to-self that
> should be optimized away in the future. Look, it's a prototype, all
> right?

I don't care about that.  What does the hybrid case look like?  Do you
prepare separate work buffers for each thread or do the threads work on
parts of a shared data structure, perhaps cooperating at higher
frequency than the MPI?  I thought you were creating strict independence
and private buffers, which would be MPI-like semantics using threads
(totally possible, and I think usually a good thing, but most threaded
programming models are explicitly trying to avoid it).

>>  If you're going to use
>> separate buffers, why is a system like MPI not sufficient?
>
> 1. No idea why you're so fixated on buffers. I've just been going for
> the simplest implementation that makes it work on this one example. It
> can all be optimized.

What would the optimized version look like?  Someone has to decide how
to index into the data structures.  You're not providing much uniformity
relative to MPI+X (e.g., X=OpenMP) if the threaded part is always sorted
out by the user in an application-dependent way.

> 2. Why is MPI not sufficient: because you have to spell out too
> much. Calculating a halo in the case of a block-cyclically distributed
> vector is way too much work. An extension of IMP would make this very
> easy to specify, and all the hard work is done under the covers.

VecScatter and PetscSF are less intrusive interfaces on top of MPI.  I
agree that MPI_Isend is pretty low level, but what are you providing
relative to these less intrusive abstractions?  You said that you were
NOT looking for 'a better notation for VecScatters", so let's assume
that such interfaces are available to the MPI programmer.

>> What semantic does your abstraction provide for hybrid
>> distributed/shared memory that imperative communication systems cannot?
>
> How about having the exact same code that I've shown you, except that
> you specify that the processes are organized as N nodes times C cores?
> Right now I've not implemented hybrid programming, but it shouldn't be
> hard.

How does the code the user writes in the callback remain the same for
MPI and threads without the copies that AMPI or shared-memory MPI would
have done and without requiring the user to explicitly deal with the
threads (working on shared data structures as with OpenMP/TBB/pthreads)?

I'm looking for a precise statement of something user code can be
ignorant of with your model, yet reap the benefits of a model where they
explicitly used that information (e.g., a semantic not possible with
shared-memory MPI/AMPI, and possible using MPI+X only with some onerous
complexity that cannot be easily tucked into a generic library
function).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140103/034c28f3/attachment.sig>

From jedbrown at mcs.anl.gov  Fri Jan  3 22:14:22 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 03 Jan 2014 22:14:22 -0600
Subject: [petsc-dev] [Jeff Hammond] Re: VPATH builds broken
Message-ID: <87y52wqt7l.fsf@jedbrown.org>

This reminds me that PETSc's build system is tied to PETSC_ARCH, thus
does not support a read-only source tree.  To do this, we would have to
make tests and Fortran stubs go into the build directory (we want that
already) and to lessen our dependence on PETSC_ARCH.  I think we should
be able to make it possible to run /path/to/petsc/configure with no
special arguments and have it set up a normal out-of-source build that
would be independent of PETSC_DIR and PETSC_ARCH.  If instead, you run
configure from PETSC_DIR, it would continue to use PETSC_ARCH as the
build directory.  (To make these otherwise uniform, PETSC_ARCH would
probably become just a hint to run make from the appropriate directory.)

Any reason we should not do this once the testing and Fortran stuff
works that way?

-------------- next part --------------
An embedded message was scrubbed...
From: Jeff Hammond <jeff.science at gmail.com>
Subject: Re: VPATH builds broken
Date: Fri, 3 Jan 2014 22:04:08 -0600
Size: 9052
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140103/3e0039b0/attachment.mht>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140103/3e0039b0/attachment.sig>

From bsmith at mcs.anl.gov  Sat Jan  4 18:01:07 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 4 Jan 2014 18:01:07 -0600
Subject: [petsc-dev] broken stuff in next please fix
Message-ID: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>


http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-freebsd-cxx-cmplx-pkgs-dbg_wii.log

/usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/bin/mpicxx   -Dpetsc_EXPORTS -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0  -fPIC    -fPIC -I/usr/home/balay/petsc.clone-3/include -I/usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/include -I/usr/local/include -o CMakeFiles/petsc.dir/src/sys/error/errtrace.c.o -c /usr/home/balay/petsc.clone-3/src/sys/error/errtrace.c
/usr/home/balay/petsc.clone-3/src/sys/error/err.c: In function 'void PetscCxxErrorThrow()':
/usr/home/balay/petsc.clone-3/src/sys/error/err.c:310: error: no matching function for call to 'std::exception::exception(const char*&)'
/usr/include/c++/4.2/exception:59: note: candidates are: std::exception::exception()
/usr/include/c++/4.2/exception:57: note:                 std::exception::exception(const std::exception&)
*** [CMakeFiles/petsc.dir/src/sys/error/err.c.o] Error code 1


http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-linux-pkgs-dbg-ftn-interfaces_crank.log

/sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning: argument type 'struct <anonymous struct at /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
1 warning generated.
/sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]

/sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:59: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
/sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:50: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
/sandbox/petsc/petsc.clone

/sandbox/petsc/petsc.clone/src/dm/impls/plex/plexsubmesh.c:2753:69: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]

/sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:328:42: warning: argument type 'struct <anonymous struct at /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:324:7> *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]



From knepley at gmail.com  Sat Jan  4 19:22:23 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 4 Jan 2014 19:22:23 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
Message-ID: <CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>

On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-freebsd-cxx-cmplx-pkgs-dbg_wii.log
>
> /usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/bin/mpicxx
> -Dpetsc_EXPORTS -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -g -O0  -fPIC    -fPIC
> -I/usr/home/balay/petsc.clone-3/include
> -I/usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/include
> -I/usr/local/include -o CMakeFiles/petsc.dir/src/sys/error/errtrace.c.o -c
> /usr/home/balay/petsc.clone-3/src/sys/error/errtrace.c
> /usr/home/balay/petsc.clone-3/src/sys/error/err.c: In function 'void
> PetscCxxErrorThrow()':
> /usr/home/balay/petsc.clone-3/src/sys/error/err.c:310: error: no matching
> function for call to 'std::exception::exception(const char*&)'
> /usr/include/c++/4.2/exception:59: note: candidates are:
> std::exception::exception()
> /usr/include/c++/4.2/exception:57: note:
> std::exception::exception(const std::exception&)
> *** [CMakeFiles/petsc.dir/src/sys/error/err.c.o] Error code 1
>
>
>
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-linux-pkgs-dbg-ftn-interfaces_crank.log
>
> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning:
> argument type 'struct <anonymous struct at
> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match
> specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *'
> [-Wtype-safety]
> 1 warning generated.
> /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning:
> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that
> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
>

Can we shut off this warning?

   Matt


> /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:59: warning:
> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that
> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
> /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:50: warning:
> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that
> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
> /sandbox/petsc/petsc.clone
>
> /sandbox/petsc/petsc.clone/src/dm/impls/plex/plexsubmesh.c:2753:69:
> warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type
> tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
>
> /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:328:42: warning:
> argument type 'struct <anonymous struct at
> /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:324:7> *' doesn't
> match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int
> *' [-Wtype-safety]
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140104/da76957f/attachment.html>

From jedbrown at mcs.anl.gov  Sat Jan  4 20:41:51 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 04 Jan 2014 20:41:51 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
Message-ID: <87ha9jqhe8.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-linux-pkgs-dbg-ftn-interfaces_crank.log
>>
>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning:
>> argument type 'struct <anonymous struct at
>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match
>> specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *'
>> [-Wtype-safety]
>> 1 warning generated.
>> /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning:
>> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that
>> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
>>
>
> Can we shut off this warning?

What version of MPICH and Clang are you using?  Current MPICH 'master'
is broken, but earlier versions with clang-3.3 are not producing this
warning.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140104/6a1c4f7c/attachment.sig>

From bsmith at mcs.anl.gov  Sat Jan  4 21:19:23 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 4 Jan 2014 21:19:23 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <87ha9jqhe8.fsf@jedbrown.org>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
Message-ID: <92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>


On Jan 4, 2014, at 8:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> 
>> On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-make_next_arch-linux-pkgs-dbg-ftn-interfaces_crank.log
>>> 
>>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning:
>>> argument type 'struct <anonymous struct at
>>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match
>>> specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *'
>>> [-Wtype-safety]
>>> 1 warning generated.
>>> /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning:
>>> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that
>>> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]
>>> 
>> 
>> Can we shut off this warning?
> 
> What version of MPICH and Clang are you using?  Current MPICH 'master'
> is broken, but earlier versions with clang-3.3 are not producing this
> warning.

    Ahh, what should we be using for nightly tests?  We can use a release or any development version. Currently we don?t seem to have a policy of what to track,.
At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz


   Barry





From jedbrown at mcs.anl.gov  Sat Jan  4 21:29:07 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 04 Jan 2014 21:29:07 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
Message-ID: <87eh4nqf7g.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>     Ahh, what should we be using for nightly tests?  We can use a release or any development version. Currently we don?t seem to have a policy of what to track,.
> At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz

Use their latest (3.0.4).  I fixed this a year ago and it is in
mpich-3.0.2 and later.

http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113de6
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140104/bd714e9b/attachment.sig>

From balay at mcs.anl.gov  Sat Jan  4 21:42:07 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Sat, 4 Jan 2014 21:42:07 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <87eh4nqf7g.fsf@jedbrown.org>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401042140480.2481@asterix>

On Sat, 4 Jan 2014, Jed Brown wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> >     Ahh, what should we be using for nightly tests?  We can use a release or any development version. Currently we don?t seem to have a policy of what to track,.
> > At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz
> 
> Use their latest (3.0.4).  I fixed this a year ago and it is in
> mpich-3.0.2 and later.
> 
> http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113de6

Barry,

We were previously using v3.0.4 [with some bug fixes for some issues that
came up in petsc testing].

http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz

Satish

From balay at mcs.anl.gov  Sat Jan  4 21:43:48 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Sat, 4 Jan 2014 21:43:48 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <alpine.LFD.2.11.1401042140480.2481@asterix>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
Message-ID: <alpine.LFD.2.11.1401042143270.2481@asterix>

On Sat, 4 Jan 2014, Satish Balay wrote:

> On Sat, 4 Jan 2014, Jed Brown wrote:
> 
> > Barry Smith <bsmith at mcs.anl.gov> writes:
> > >     Ahh, what should we be using for nightly tests?  We can use a release or any development version. Currently we don?t seem to have a policy of what to track,.
> > > At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz
> > 
> > Use their latest (3.0.4).  I fixed this a year ago and it is in
> > mpich-3.0.2 and later.
> > 
> > http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113de6
> 
> Barry,
> 
> We were previously using v3.0.4 [with some bug fixes for some issues that
> came up in petsc testing].
> 
> http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz

BTW: This is the version used in 'maint'/petsc-3.4 aswell.

Satish

From jedbrown at mcs.anl.gov  Sat Jan  4 21:49:10 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 04 Jan 2014 21:49:10 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <alpine.LFD.2.11.1401042140480.2481@asterix>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
Message-ID: <87bnzrqea1.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> We were previously using v3.0.4 [with some bug fixes for some issues that
> came up in petsc testing].
>
> http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz

Barry, what was the idea here?

https://bitbucket.org/petsc/petsc/commits/7904a3320b8a1f45505a06da988fac624ac5e24c

It's certainly unrelated to the rest of the commit.  Shall we just
revert that part?


And how is it that the MPICH release has a memory bug for 8 months
during which time they haven't put out a new patch release (3.0.5)?

http://git.mpich.org/mpich.git/commit/3adb59cb83a4f675ba990062c62396d1a23483cf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140104/ee80bcb7/attachment.sig>

From john.mousel at gmail.com  Sun Jan  5 11:46:45 2014
From: john.mousel at gmail.com (John Mousel)
Date: Sun, 5 Jan 2014 11:46:45 -0600
Subject: [petsc-dev] ParMETIS assertion with pc_gamg_repartition
Message-ID: <CA+zegpkv6cEbO6dro-CBMGVkrmoQOmCnCzwimuXiU0rei_VXxA@mail.gmail.com>

I'm getting a ParMETIS assertion:

[53] ***ASSERTION failed on line 176 of file
/nics/b/home/jmousel/NumericalLibraries/petsc/intel-debug/externalpackages/parmetis-4.0.3-p1/libparmetis/comm.c:
j == nnbrs
[110] ***ASSERTION failed on line 176 of file
/nics/b/home/jmousel/NumericalLibraries/petsc/intel-debug/externalpackages/parmetis-4.0.3-p1/libparmetis/comm.c:
j == nnbrs

when I add -redistribute_pc_gamg_repartition true to my option list:

 -pres_ksp_type preonly -pres_pc_type redistribute
-pres_redistribute_ksp_type pgmres -pres_redistribute_pc_type gamg
-pres_redistribute_pc_gamg_threshold 0.05
-pres_redistribute_mg_levels_ksp_type richardson
-pres_redistribute_mg_levels_pc_type sor
-pres_redistribute_mg_coarse_ksp_type richardson
-pres_redistribute_mg_coarse_pc_type sor
-pres_redistribute_mg_coarse_pc_sor_its 3 -pres_redistribute_mg_type full
-pres_redistribute_pc_gamg_type agg -pres_redistribute_pc_gamg_agg_nsmooths
1 -pres_redistribute_pc_gamg_sym_graph true -ELAFINT_flex_comm_size 16
-vel_ksp_type ibcgs -vel_ksp_monitor -pres_redistribute_ksp_monitor
-lspaint_ksp_monitor -pres_redistribute_pc_gamg_repartition true
-pres_redistribute_pc_gamg_reuse_interpolation

There is no additional stack trace to provide. I receive the same message
for 12, 48, 60, and 120 cores. The matrix has about 12 million rows.


John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140105/78ed3f4d/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan  5 12:49:57 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 5 Jan 2014 12:49:57 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <87bnzrqea1.fsf@jedbrown.org>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
	<87bnzrqea1.fsf@jedbrown.org>
Message-ID: <09C92675-7F67-419B-B0A1-DB2A0A839266@mcs.anl.gov>


   I have switched back to using the mpich-master-v3.0.4-106-g3adb59c.tar.gz   I messed up and put that change into a commit when I shouldn?t have. (I still miss the nice bk gui that showed cleanly each change you were committing).

   But it is extremely worrisome that we need to use PARTICULAR versions of MPI implementations to get clean compilers?!?!?!

   1) Isn?t this fundamentally wrong? 

   1.5) Do we want to be in the business of saying, ?if you use compiler XX version YY then you need to use MPICH implementation version ZZ, otherwise it won?t work?

   2) Shouldn?t we be portable to all the various MPI implementations that are released at various times? 

   3) Do we need a configure test to properly handle the ?broken? MPI implementations? 

   It seems to me that we need to have configure either

   A) declare certain MPI implementations as broken and tell the user to change implementations or

   B) use conditional compiles so PETSc builds cleanly even with these ?broken? MPI implementations.

   How are we going to fix this? remember not everyone uses ?download-mpich

   Barry



On Jan 4, 2014, at 9:49 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
>> We were previously using v3.0.4 [with some bug fixes for some issues that
>> came up in petsc testing].
>> 
>> http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz
> 
> Barry, what was the idea here?
> 
> https://bitbucket.org/petsc/petsc/commits/7904a3320b8a1f45505a06da988fac624ac5e24c
> 
> It's certainly unrelated to the rest of the commit.  Shall we just
> revert that part?
> 
> 
> And how is it that the MPICH release has a memory bug for 8 months
> during which time they haven't put out a new patch release (3.0.5)?
> 
> http://git.mpich.org/mpich.git/commit/3adb59cb83a4f675ba990062c62396d1a23483cf



From jedbrown at mcs.anl.gov  Sun Jan  5 13:09:57 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 05 Jan 2014 13:09:57 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <09C92675-7F67-419B-B0A1-DB2A0A839266@mcs.anl.gov>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
	<87bnzrqea1.fsf@jedbrown.org>
	<09C92675-7F67-419B-B0A1-DB2A0A839266@mcs.anl.gov>
Message-ID: <87wqiep7ne.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    I have switched back to using the
>    mpich-master-v3.0.4-106-g3adb59c.tar.gz I messed up and put that
>    change into a commit when I shouldn?t have. (I still miss the nice
>    bk gui that showed cleanly each change you were committing).

I never used the bk gui, but what is lacking in "git gui" or sourcetree?

>    But it is extremely worrisome that we need to use PARTICULAR
>    versions of MPI implementations to get clean compilers?!?!?!
>
>    1) Isn?t this fundamentally wrong? 

Like all dependencies, MPI implementations can have bugs.  This was a
bug and was fixed before it spent much time in the wild.  The memory bug
is more subtle and I have no idea why mpich-3.0.5 has not been released.

>    1.5) Do we want to be in the business of saying, ?if you use
>    compiler XX version YY then you need to use MPICH implementation
>    version ZZ, otherwise it won?t work?

This is just a warning, but some MPI implementations have functions that
crash at runtime.  Others have space leaks (e.g., V1R2M0 on BG/Q).

>    2) Shouldn?t we be portable to all the various MPI implementations that are released at various times? 
>
>    3) Do we need a configure test to properly handle the ?broken? MPI implementations? 

Lots of compilers have bugs too, including wrong code.  I don't think
it's maintainable to catalog all the bugs in all dependencies on all
architectures.  If something is buggy on a popular system, such that it
creates complaints, we can work around it.  Note that some packages like
ParMETIS have major open bugs that people hit all the time, and we
maintain patches that upstream inexplicably does not apply.

In the case of MPI type tags, the user encounters the same warnings if
they use MPI_2INT.


Public shaming for known bugs that intentionally cannot be detected at
configure time is one marginally-effective recourse.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140105/c82b9a70/attachment.sig>

From bsmith at mcs.anl.gov  Sun Jan  5 13:15:50 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 5 Jan 2014 13:15:50 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <87wqiep7ne.fsf@jedbrown.org>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
	<87bnzrqea1.fsf@jedbrown.org>
	<09C92675-7F67-419B-B0A1-DB2A0A839266@mcs.anl.gov>
	<87wqiep7ne.fsf@jedbrown.org>
Message-ID: <4D2BA054-52C8-4C05-AF0D-4DEC7D27CB0A@mcs.anl.gov>


On Jan 5, 2014, at 1:09 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>   I have switched back to using the
>>   mpich-master-v3.0.4-106-g3adb59c.tar.gz I messed up and put that
>>   change into a commit when I shouldn?t have. (I still miss the nice
>>   bk gui that showed cleanly each change you were committing).
> 
> I never used the bk gui, but what is lacking in "git gui" or sourcetree?
> 
>>   But it is extremely worrisome that we need to use PARTICULAR
>>   versions of MPI implementations to get clean compilers?!?!?!
>> 
>>   1) Isn?t this fundamentally wrong? 
> 
> Like all dependencies, MPI implementations can have bugs.  This was a
> bug and was fixed before it spent much time in the wild.  

    It appears that this bug appeared in a release and hence could affect many users.  I think configure should detect it and do either A) or B) otherwise it looks to all users like a bug in PETSc, when it is not.  Yes we cannot catalog all bugs but ones that we know about and that make PETSc look bad should be handled.
 

   Barry


> The memory bug
> is more subtle and I have no idea why mpich-3.0.5 has not been released.
> 
>>   1.5) Do we want to be in the business of saying, ?if you use
>>   compiler XX version YY then you need to use MPICH implementation
>>   version ZZ, otherwise it won?t work?
> 
> This is just a warning, but some MPI implementations have functions that
> crash at runtime.  Others have space leaks (e.g., V1R2M0 on BG/Q).
> 
>>   2) Shouldn?t we be portable to all the various MPI implementations that are released at various times? 
>> 
>>   3) Do we need a configure test to properly handle the ?broken? MPI implementations? 
> 
> Lots of compilers have bugs too, including wrong code.  I don't think
> it's maintainable to catalog all the bugs in all dependencies on all
> architectures.  If something is buggy on a popular system, such that it
> creates complaints, we can work around it.  Note that some packages like
> ParMETIS have major open bugs that people hit all the time, and we
> maintain patches that upstream inexplicably does not apply.
> 
> In the case of MPI type tags, the user encounters the same warnings if
> they use MPI_2INT.
> 
> 
> Public shaming for known bugs that intentionally cannot be detected at
> configure time is one marginally-effective recourse.



From jedbrown at mcs.anl.gov  Sun Jan  5 13:31:43 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 05 Jan 2014 13:31:43 -0600
Subject: [petsc-dev] broken stuff in next please fix
In-Reply-To: <4D2BA054-52C8-4C05-AF0D-4DEC7D27CB0A@mcs.anl.gov>
References: <278FF694-78F2-4590-BF8F-7B810E3F479A@mcs.anl.gov>
	<CAMYG4GmgO4_1dfYH7VZTVkjiBRqAuQ5LaCVxL+kfKSgPLechxg@mail.gmail.com>
	<87ha9jqhe8.fsf@jedbrown.org>
	<92DE3EEE-F511-4803-85DE-D80153926029@mcs.anl.gov>
	<87eh4nqf7g.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401042140480.2481@asterix>
	<87bnzrqea1.fsf@jedbrown.org>
	<09C92675-7F67-419B-B0A1-DB2A0A839266@mcs.anl.gov>
	<87wqiep7ne.fsf@jedbrown.org>
	<4D2BA054-52C8-4C05-AF0D-4DEC7D27CB0A@mcs.anl.gov>
Message-ID: <87txdip6n4.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>     It appears that this bug appeared in a release and hence could
>     affect many users.  

Same with ParMETIS bugs, V1R2M0, and numerous compiler bugs.  PETSc
releases have bugs too.  Releases are never completely bug-free;
responsible projects put out patch releases quickly after finding bugs,
and make it as easy as possible to upgrade.

>     I think configure should detect it and do either A) or B)
>     otherwise it looks to all users like a bug in PETSc, when it is
>     not.  Yes we cannot catalog all bugs but ones that we know about
>     and that make PETSc look bad should be handled.

If we do this with all known bugs, BuildSystem will grow to be bigger
than PETSc and take hours to run.  mpich-3.0.4 has been out for a long
time now, and most everyone with a sensible distribution has had a
painless and automatic upgrade.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140105/578c2f2c/attachment.sig>

From balay at mcs.anl.gov  Mon Jan  6 14:59:35 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 14:59:35 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401061452300.17785@asterix>

Barry,

Do you want this for only 'manpages' or all generated html?

if all generated docs - then this would be same as adding as the
version/date on the top-right corner.

If only manpages - then I'll have to add this to a location where the
manpages are generated/modified.

thanks,
Satish

On Wed, 1 Jan 2014, Barry Smith wrote:

> 
>   Satish,
> 
>     Resending this. Should be really straightforward, please let me know if there are difficulties in getting this done soon.
> 
>     Thanks
> 
>      Barry
> 
> Request-assigned: balay add Report Typos and Errors link to all manual pages
> 
> On Dec 10, 2013, at 11:11 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > 
> >   Satish,
> > 
> >    Please add to the generation of all manual pages (in the upper right corner) a Report Typos and Errors link using mailto:  For example something like
> > 
> > <a href=?mailto:petsc-maint at mcs.anl.gov?subject=Typo or Error in MANUALPAGE &body=Please describe the typo or error in the manual page">Report Typos and Errors</a>
> > 
> > where MANUALPAGE is the name of that generated manual page.
> > 
> >  Thanks
> > 
> >   Barry
> > 
> 
> 

From jedbrown at mcs.anl.gov  Mon Jan  6 15:04:33 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 15:04:33 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061452300.17785@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
Message-ID: <87ob3oizz2.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> Do you want this for only 'manpages' or all generated html?

I want it (somewhere unintrusive) for all HTML pages (generated or not).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/78c9da2a/attachment.sig>

From balay at mcs.anl.gov  Mon Jan  6 15:11:49 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 15:11:49 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87ob3oizz2.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061508310.17785@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> > Do you want this for only 'manpages' or all generated html?
> 
> I want it (somewhere unintrusive) for all HTML pages (generated or not).
> 

How about: http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html

[perhaps 'MANUALPAGE' can be changed to 'documentation' - if we are doing this for all docs.]

Satish

diff --git a/makefile b/makefile
index 362c004..9de2ad2 100644
--- a/makefile
+++ b/makefile
@@ -335,11 +335,13 @@ docsetdate: chk_petscdir
         fi; \
         datestr=`git log -1 --pretty=format:%ci | cut -d ' ' -f 1`; \
         export datestr; \
+        typostr='   <div id="bugreport" align=right><a href="mailto:petsc-maint at mcs.anl.gov?subject=Typo or Error in MANUALPAGE &body=Please describe the typo or error in the manual page">Report Typos and Errors</a></div>';\
+        export typostr; \
         find * -type d -wholename src/docs/website -prune -o -type d -wholename src/benchmarks/results -prune -o \
           -type d -wholename config/BuildSystem/docs/website -prune -o -type d -wholename include/web -prune -o \
           -type d -wholename 'arch-*' -prune -o -type d -wholename src/tops -prune -o -type d -wholename externalpackages -prune -o \
           -type f -wholename tutorials/multiphysics/tutorial.html -prune -o -type f -name \*.html \
-          -exec perl -pi -e 's^(<body.*>)^$$1\n   <div id=\"version\" align=right><b>$$ENV{petscversion} $$ENV{datestr}</b></div>^i' {} \; \
+          -exec perl -pi -e 's^(<body.*>)^$$1\n   <div id=\"version\" align=right><b>$$ENV{petscversion} $$ENV{datestr}</b></div>\n$$ENV{typostr}^i' {} \; \
           -exec perl -pi -e 's^(<head>)^$$1 <link rel="canonical" href="http://www.mcs.anl.gov/petsc/petsc-current/{}" />^i' {} \; ; \
         echo "Done fixing version number, date, canonical URL info"


From jedbrown at mcs.anl.gov  Mon Jan  6 15:18:06 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 15:18:06 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061508310.17785@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
Message-ID: <87iotwizch.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> How about: http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html
>
> [perhaps 'MANUALPAGE' can be changed to 'documentation' - if we are doing this for all docs.]

The message should come preformatted to include the URL of the page on
which a typo is being reported.  I would also reduce the font size a
little.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/db0a526f/attachment.sig>

From balay at mcs.anl.gov  Mon Jan  6 15:53:03 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 15:53:03 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87iotwizch.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061548270.17785@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > How about: http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html
> >
> > [perhaps 'MANUALPAGE' can be changed to 'documentation' - if we are doing this for all docs.]
> 
> The message should come preformatted to include the URL of the page on
> which a typo is being reported.  I would also reduce the font size a
> little.

I can add in <small> formatting for the text.

But wrt URL - since we don't know where the docs will land - I can add
in the following context to the message:

Please describe the typo or error in the documentation: petsc-VER GIT-ID FILE-NAME

i.e

Please describe the typo or error in the documentation: petsc-dev 06283fd4323cef45a7147b2226c8e0c084e2a1d2 docs/manualpages/Vec/VecView.html

http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html is updated

Satish


------------

$ git diff |cat
diff --git a/makefile b/makefile
index 362c004..44d50cf 100644
--- a/makefile
+++ b/makefile
@@ -335,11 +335,13 @@ docsetdate: chk_petscdir
         fi; \
         datestr=`git log -1 --pretty=format:%ci | cut -d ' ' -f 1`; \
         export datestr; \
+        gitver=`git log -1 --pretty=format:%H`; \
+        export gitver; \
         find * -type d -wholename src/docs/website -prune -o -type d -wholename src/benchmarks/results -prune -o \
           -type d -wholename config/BuildSystem/docs/website -prune -o -type d -wholename include/web -prune -o \
           -type d -wholename 'arch-*' -prune -o -type d -wholename src/tops -prune -o -type d -wholename externalpackages -prune -o \
           -type f -wholename tutorials/multiphysics/tutorial.html -prune -o -type f -name \*.html \
-          -exec perl -pi -e 's^(<body.*>)^$$1\n   <div id=\"version\" align=right><b>$$ENV{petscversion} $$ENV{datestr}</b></div>^i' {} \; \
+          -exec perl -pi -e 's^(<body.*>)^$$1\n   <div id=\"version\" align=right><b>$$ENV{petscversion} $$ENV{datestr}</b></div>\n   <div id="bugreport" align=right><a href="mailto:petsc-maint at mcs.anl.gov?subject=Typo or Error in Documentation &body=Please describe the typo or error in the documentation: $$ENV{petscversion} $$ENV{gitver} {} "><small>Report Typos and Errors</small></a></div>^i' {} \; \
           -exec perl -pi -e 's^(<head>)^$$1 <link rel="canonical" href="http://www.mcs.anl.gov/petsc/petsc-current/{}" />^i' {} \; ; \
         echo "Done fixing version number, date, canonical URL info"


From jedbrown at mcs.anl.gov  Mon Jan  6 15:59:10 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 15:59:10 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061548270.17785@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
Message-ID: <878uusixg1.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> But wrt URL - since we don't know where the docs will land - I can add
> in the following context to the message:
>
> Please describe the typo or error in the documentation: petsc-VER GIT-ID FILE-NAME
>
> i.e
>
> Please describe the typo or error in the documentation: petsc-dev 06283fd4323cef45a7147b2226c8e0c084e2a1d2 docs/manualpages/Vec/VecView.html
>
> http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html is updated

Looks good to me.

> $ git diff |cat
> diff --git a/makefile b/makefile
> index 362c004..44d50cf 100644
> --- a/makefile
> +++ b/makefile
> @@ -335,11 +335,13 @@ docsetdate: chk_petscdir
>          fi; \
>          datestr=`git log -1 --pretty=format:%ci | cut -d ' ' -f 1`; \
>          export datestr; \
> +        gitver=`git log -1 --pretty=format:%H`; \

I would use "git describe" instead since it has some human-readable
context.

  v3.4.3-2283-g06283fd
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/fc74d290/attachment.sig>

From wdn at lanl.gov  Mon Jan  6 16:07:18 2014
From: wdn at lanl.gov (Nystrom, William D)
Date: Mon, 6 Jan 2014 22:07:18 +0000
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <878uusixg1.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>,
	<878uusixg1.fsf@jedbrown.org>
Message-ID: <CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>

When I click on your URL and then click on "Report Typos and Errors", I get a window
that pops up to get me to setup evolution.  But I don't use evolution.  Not sure what
would happen on some other OS like Windows or Mac OS.  Also, I often have popups
disabled.

Is it possible to make this work without the above issues?

Dave

--
Dave Nystrom
LANL HPC-5
Phone: 505-667-7913
Email: wdn at lanl.gov
Smail: Mail Stop B272
       Group HPC-5
       Los Alamos National Laboratory
       Los Alamos, NM 87545


________________________________________
From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
Sent: Monday, January 06, 2014 2:59 PM
To: Satish Balay
Cc: petsc-dev
Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages

Satish Balay <balay at mcs.anl.gov> writes:

> But wrt URL - since we don't know where the docs will land - I can add
> in the following context to the message:
>
> Please describe the typo or error in the documentation: petsc-VER GIT-ID FILE-NAME
>
> i.e
>
> Please describe the typo or error in the documentation: petsc-dev 06283fd4323cef45a7147b2226c8e0c084e2a1d2 docs/manualpages/Vec/VecView.html
>
> http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html is updated

Looks good to me.

> $ git diff |cat
> diff --git a/makefile b/makefile
> index 362c004..44d50cf 100644
> --- a/makefile
> +++ b/makefile
> @@ -335,11 +335,13 @@ docsetdate: chk_petscdir
>          fi; \
>          datestr=`git log -1 --pretty=format:%ci | cut -d ' ' -f 1`; \
>          export datestr; \
> +        gitver=`git log -1 --pretty=format:%H`; \

I would use "git describe" instead since it has some human-readable
context.

  v3.4.3-2283-g06283fd


From balay at mcs.anl.gov  Mon Jan  6 16:09:29 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 16:09:29 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <878uusixg1.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061604310.17785@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > But wrt URL - since we don't know where the docs will land - I can add
> > in the following context to the message:
> >
> > Please describe the typo or error in the documentation: petsc-VER GIT-ID FILE-NAME
> >
> > i.e
> >
> > Please describe the typo or error in the documentation: petsc-dev 06283fd4323cef45a7147b2226c8e0c084e2a1d2 docs/manualpages/Vec/VecView.html
> >
> > http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html is updated
> 
> Looks good to me.
> 
> > $ git diff |cat
> > diff --git a/makefile b/makefile
> > index 362c004..44d50cf 100644
> > --- a/makefile
> > +++ b/makefile
> > @@ -335,11 +335,13 @@ docsetdate: chk_petscdir
> >          fi; \
> >          datestr=`git log -1 --pretty=format:%ci | cut -d ' ' -f 1`; \
> >          export datestr; \
> > +        gitver=`git log -1 --pretty=format:%H`; \
> 
> I would use "git describe" instead since it has some human-readable
> context.
> 
>   v3.4.3-2283-g06283fd

Hm - we use 06283fd4323cef45a7147b2226c8e0c084e2 notation for
PETSC_VERSION_GIT in petscconf.h [via configure] and in petscversion.h
[during tarball generation]

Should these also be changed to the new notation?

Satish


From jedbrown at mcs.anl.gov  Mon Jan  6 16:15:33 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 16:15:33 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
Message-ID: <8761pwiwoq.fsf@jedbrown.org>

"Nystrom, William D" <wdn at lanl.gov> writes:

> When I click on your URL and then click on "Report Typos and Errors", I get a window
> that pops up to get me to setup evolution.  But I don't use evolution.  

This is the same thing that happens if you click any mailto link in your
browser.  You can change the default email client.  It seems strange
that this configuration tidbit would still be a problem, but a more
braindead alternative is to do a form submission instead of email (the
form submission can just send email to petsc-maint, though it probably
needs a CAPTCHA).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/98b23efb/attachment.sig>

From jedbrown at mcs.anl.gov  Mon Jan  6 16:17:34 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 16:17:34 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061604310.17785@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061604310.17785@asterix>
Message-ID: <8738l0iwld.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> Hm - we use 06283fd4323cef45a7147b2226c8e0c084e2 notation for
> PETSC_VERSION_GIT in petscconf.h [via configure] and in petscversion.h
> [during tarball generation]
>
> Should these also be changed to the new notation?

I think I suggested that a long time ago and would mildly prefer it now
too, though those uses have accompanying data that provides much of the
contextual information in git describe.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/21abadd9/attachment.sig>

From balay at mcs.anl.gov  Mon Jan  6 16:18:42 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 16:18:42 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <8761pwiwoq.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061616480.17785@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> "Nystrom, William D" <wdn at lanl.gov> writes:
> 
> > When I click on your URL and then click on "Report Typos and Errors", I get a window
> > that pops up to get me to setup evolution.  But I don't use evolution.  
> 
> This is the same thing that happens if you click any mailto link in your
> browser.  You can change the default email client.  It seems strange
> that this configuration tidbit would still be a problem, but a more
> braindead alternative is to do a form submission instead of email (the
> form submission can just send email to petsc-maint, though it probably
> needs a CAPTCHA).

I supporse each web browser has its own procedure for configuring the default mail client.

firefox provides the following doc to setup the default e-mail client.

https://support.mozilla.org/en-US/kb/change-program-used-open-email-links

But I guess - it doesn't support my prefered mail clinet: alpine.

Satish





From aron at ahmadia.net  Mon Jan  6 16:19:41 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Mon, 6 Jan 2014 17:19:41 -0500
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <8761pwiwoq.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
Message-ID: <CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>

Perhaps make the Email Address visible in the URL link?

"Click here to report typos/errors to petsc-maint at mcs.anl.gov".

Make the entire statement a mailto link?  Then anybody who gets in a
confusing mailto situation will at least know what the URL was supposed to
do (open a bug report).


On Mon, Jan 6, 2014 at 5:15 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> "Nystrom, William D" <wdn at lanl.gov> writes:
>
> > When I click on your URL and then click on "Report Typos and Errors", I
> get a window
> > that pops up to get me to setup evolution.  But I don't use evolution.
>
> This is the same thing that happens if you click any mailto link in your
> browser.  You can change the default email client.  It seems strange
> that this configuration tidbit would still be a problem, but a more
> braindead alternative is to do a form submission instead of email (the
> form submission can just send email to petsc-maint, though it probably
> needs a CAPTCHA).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/f49b56cd/attachment.html>

From jedbrown at mcs.anl.gov  Mon Jan  6 16:25:41 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 16:25:41 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
Message-ID: <87zjn8hhne.fsf@jedbrown.org>

Aron Ahmadia <aron at ahmadia.net> writes:

> Perhaps make the Email Address visible in the URL link?
>
> "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
>
> Make the entire statement a mailto link?  Then anybody who gets in a
> confusing mailto situation will at least know what the URL was supposed to
> do (open a bug report).

Good call, but lose the fluff:

  "report errors to petsc-maint at mcs.anl.gov"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/33e68b59/attachment.sig>

From wdn at lanl.gov  Mon Jan  6 16:26:34 2014
From: wdn at lanl.gov (Nystrom, William D)
Date: Mon, 6 Jan 2014 22:26:34 +0000
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061616480.17785@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>,
	<alpine.LFD.2.11.1401061616480.17785@asterix>
Message-ID: <CB90CF2315BF6A44986ACD4A47418D226F151DBB@ECS-EXG-P-MB05.win.lanl.gov>

And I often use xemacs vm for my email client.

I wonder if the form submission would be more robust.  Many users might just
punt on submitting the typo/error info if they have to go configure something
on their system.

________________________________________
From: Satish Balay [balay at mcs.anl.gov]
Sent: Monday, January 06, 2014 3:18 PM
To: Jed Brown
Cc: Nystrom, William D; petsc-dev
Subject: RE: [petsc-dev] adding Report Typo and Errors link to manual pages

On Mon, 6 Jan 2014, Jed Brown wrote:

> "Nystrom, William D" <wdn at lanl.gov> writes:
>
> > When I click on your URL and then click on "Report Typos and Errors", I get a window
> > that pops up to get me to setup evolution.  But I don't use evolution.
>
> This is the same thing that happens if you click any mailto link in your
> browser.  You can change the default email client.  It seems strange
> that this configuration tidbit would still be a problem, but a more
> braindead alternative is to do a form submission instead of email (the
> form submission can just send email to petsc-maint, though it probably
> needs a CAPTCHA).

I supporse each web browser has its own procedure for configuring the default mail client.

firefox provides the following doc to setup the default e-mail client.

https://support.mozilla.org/en-US/kb/change-program-used-open-email-links

But I guess - it doesn't support my prefered mail clinet: alpine.

Satish





From wdn at lanl.gov  Mon Jan  6 16:29:52 2014
From: wdn at lanl.gov (Nystrom, William D)
Date: Mon, 6 Jan 2014 22:29:52 +0000
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87zjn8hhne.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
Message-ID: <CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>

But if the user has to submit the error info by emailing manually, would
you likely lose some of the info you are trying to capture automatically?

________________________________________
From: Jed Brown [jedbrown at mcs.anl.gov]
Sent: Monday, January 06, 2014 3:25 PM
To: Aron Ahmadia
Cc: Nystrom, William D; Satish Balay; petsc-dev
Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages

Aron Ahmadia <aron at ahmadia.net> writes:

> Perhaps make the Email Address visible in the URL link?
>
> "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
>
> Make the entire statement a mailto link?  Then anybody who gets in a
> confusing mailto situation will at least know what the URL was supposed to
> do (open a bug report).

Good call, but lose the fluff:

  "report errors to petsc-maint at mcs.anl.gov"


From balay at mcs.anl.gov  Mon Jan  6 16:52:22 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 16:52:22 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
Message-ID: <alpine.LFD.2.11.1401061649460.3067@asterix>

so 'mailto' is no good.

and just adding text "report errors to petsc-maint at mcs.anl.gov"
doesn't improve on status-quo.

so 'web submission form' would be the prefered thingy? I'll have to
check if thats feasiable..

Satish

On Mon, 6 Jan 2014, Nystrom, William D wrote:

> But if the user has to submit the error info by emailing manually, would
> you likely lose some of the info you are trying to capture automatically?
> 
> ________________________________________
> From: Jed Brown [jedbrown at mcs.anl.gov]
> Sent: Monday, January 06, 2014 3:25 PM
> To: Aron Ahmadia
> Cc: Nystrom, William D; Satish Balay; petsc-dev
> Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages
> 
> Aron Ahmadia <aron at ahmadia.net> writes:
> 
> > Perhaps make the Email Address visible in the URL link?
> >
> > "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
> >
> > Make the entire statement a mailto link?  Then anybody who gets in a
> > confusing mailto situation will at least know what the URL was supposed to
> > do (open a bug report).
> 
> Good call, but lose the fluff:
> 
>   "report errors to petsc-maint at mcs.anl.gov"
> 



From bsmith at mcs.anl.gov  Mon Jan  6 17:53:13 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 6 Jan 2014 17:53:13 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061649460.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
Message-ID: <2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>


On Jan 6, 2014, at 4:52 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> so 'mailto' is no good.

   Not everyone lives in a cave and uses the pony-express to communicate. Are really most PETSc users and developers complete troglodytes?

   Note that you can setup mailto links to go to Emacs https://code.google.com/p/plume-lib/source/browse/bin/emacs-mailto-handler?name=v1.0.0 and alpine https://groups.google.com/forum/#!topic/comp.mail.pine/puiLTBBQDBg

   Satish,

   I like the mailto: link you have set up and would like it to stay. It looks great to me but it currently doesn?t do the address correctly: it gives me petsc-maint.anl.gov
when it should give petsc-maint at mcs.anl.gov

   Put it into next and master

   Thanks

   Barry

   Nix the form web submission form idea, MCS has such a primitive web server system it will suck.



> 
> and just adding text "report errors to petsc-maint at mcs.anl.gov"
> doesn't improve on status-quo.
> 
> so 'web submission form' would be the prefered thingy? I'll have to
> check if thats feasiable..
> 
> Satish
> 
> On Mon, 6 Jan 2014, Nystrom, William D wrote:
> 
>> But if the user has to submit the error info by emailing manually, would
>> you likely lose some of the info you are trying to capture automatically?
>> 
>> ________________________________________
>> From: Jed Brown [jedbrown at mcs.anl.gov]
>> Sent: Monday, January 06, 2014 3:25 PM
>> To: Aron Ahmadia
>> Cc: Nystrom, William D; Satish Balay; petsc-dev
>> Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages
>> 
>> Aron Ahmadia <aron at ahmadia.net> writes:
>> 
>>> Perhaps make the Email Address visible in the URL link?
>>> 
>>> "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
>>> 
>>> Make the entire statement a mailto link?  Then anybody who gets in a
>>> confusing mailto situation will at least know what the URL was supposed to
>>> do (open a bug report).
>> 
>> Good call, but lose the fluff:
>> 
>>  "report errors to petsc-maint at mcs.anl.gov"
>> 
> 



From balay at mcs.anl.gov  Mon Jan  6 18:36:14 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 18:36:14 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401061833150.3067@asterix>

On Mon, 6 Jan 2014, Barry Smith wrote:

> 
> On Jan 6, 2014, at 4:52 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > so 'mailto' is no good.
> 
>    Not everyone lives in a cave and uses the pony-express to communicate. Are really most PETSc users and developers complete troglodytes?
>

Well - most don't have mailto: configured from their web browser [eventhough they might use the spiffer new mail clients]


I guess thats a one time setup cost though..

>    Note that you can setup mailto links to go to Emacs https://code.google.com/p/plume-lib/source/browse/bin/emacs-mailto-handler?name=v1.0.0 and alpine https://groups.google.com/forum/#!topic/comp.mail.pine/puiLTBBQDBg
> 
>    Satish,
> 
>    I like the mailto: link you have set up and would like it to stay. It looks great to me but it currently doesn?t do the address correctly: it gives me petsc-maint.anl.gov
> when it should give petsc-maint at mcs.anl.gov

fixed.

> 
>    Put it into next and master

ok merged to next for now..

satish

> 
>    Thanks
> 
>    Barry
> 
>    Nix the form web submission form idea, MCS has such a primitive web server system it will suck.
> 
> 
> 
> > 
> > and just adding text "report errors to petsc-maint at mcs.anl.gov"
> > doesn't improve on status-quo.
> > 
> > so 'web submission form' would be the prefered thingy? I'll have to
> > check if thats feasiable..
> > 
> > Satish
> > 
> > On Mon, 6 Jan 2014, Nystrom, William D wrote:
> > 
> >> But if the user has to submit the error info by emailing manually, would
> >> you likely lose some of the info you are trying to capture automatically?
> >> 
> >> ________________________________________
> >> From: Jed Brown [jedbrown at mcs.anl.gov]
> >> Sent: Monday, January 06, 2014 3:25 PM
> >> To: Aron Ahmadia
> >> Cc: Nystrom, William D; Satish Balay; petsc-dev
> >> Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages
> >> 
> >> Aron Ahmadia <aron at ahmadia.net> writes:
> >> 
> >>> Perhaps make the Email Address visible in the URL link?
> >>> 
> >>> "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
> >>> 
> >>> Make the entire statement a mailto link?  Then anybody who gets in a
> >>> confusing mailto situation will at least know what the URL was supposed to
> >>> do (open a bug report).
> >> 
> >> Good call, but lose the fluff:
> >> 
> >>  "report errors to petsc-maint at mcs.anl.gov"
> >> 
> > 
> 
> 

From balay at mcs.anl.gov  Mon Jan  6 18:37:48 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 18:37:48 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <8738l0iwld.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061604310.17785@asterix>
	<8738l0iwld.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061836190.3067@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> > Hm - we use 06283fd4323cef45a7147b2226c8e0c084e2 notation for
> > PETSC_VERSION_GIT in petscconf.h [via configure] and in petscversion.h
> > [during tarball generation]
> >
> > Should these also be changed to the new notation?
> 
> I think I suggested that a long time ago and would mildly prefer it now
> too, though those uses have accompanying data that provides much of the
> contextual information in git describe.

One can always derive the readable info from the full hash..

$ git describe 06283fd4323cef45a7147b2226c8e0c084e2
v3.4.3-2283-g06283fd

For now I'll keep the full hash here. If it needs to be changed - all
usages can be changed at the same time..

Satish


From balay at mcs.anl.gov  Mon Jan  6 18:39:09 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 18:39:09 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87zjn8hhne.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401061838000.3067@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Aron Ahmadia <aron at ahmadia.net> writes:
> 
> > Perhaps make the Email Address visible in the URL link?
> >
> > "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
> >
> > Make the entire statement a mailto link?  Then anybody who gets in a
> > confusing mailto situation will at least know what the URL was supposed to
> > do (open a bug report).
> 
> Good call, but lose the fluff:
> 
>   "report errors to petsc-maint at mcs.anl.gov"

With the mailto: - the above is a bit superflus..  Browsers have a way
to copy the e-mail-id from the link [which I usually use..]

Satish


From jedbrown at mcs.anl.gov  Mon Jan  6 18:48:38 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 06 Jan 2014 18:48:38 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061836190.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061604310.17785@asterix>
	<8738l0iwld.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061836190.3067@asterix>
Message-ID: <87r48khb15.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> One can always derive the readable info from the full hash..
>
> $ git describe 06283fd4323cef45a7147b2226c8e0c084e2
> v3.4.3-2283-g06283fd
>
> For now I'll keep the full hash here. If it needs to be changed - all
> usages can be changed at the same time..

The reason I prefer the "git describe" output is that it provides the
information up-front, so that you can have it in mind while you think
about what might be wrong.  For example, if we see v2.2.1-54321-g01dbad,
we know right off that someone must have copied ancient pages somewhere.
The cost of manual translation is not just the time to perform the
mechanical steps, but also the time spent thinking about the problem
without the benefit of that information.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140106/c1ec0e9c/attachment.sig>

From balay at mcs.anl.gov  Mon Jan  6 20:01:01 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 6 Jan 2014 20:01:01 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87r48khb15.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061604310.17785@asterix>
	<8738l0iwld.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061836190.3067@asterix>
	<87r48khb15.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401062000430.3067@asterix>

On Mon, 6 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > One can always derive the readable info from the full hash..
> >
> > $ git describe 06283fd4323cef45a7147b2226c8e0c084e2
> > v3.4.3-2283-g06283fd
> >
> > For now I'll keep the full hash here. If it needs to be changed - all
> > usages can be changed at the same time..
> 
> The reason I prefer the "git describe" output is that it provides the
> information up-front, so that you can have it in mind while you think
> about what might be wrong.  For example, if we see v2.2.1-54321-g01dbad,
> we know right off that someone must have copied ancient pages somewhere.
> The cost of manual translation is not just the time to perform the
> mechanical steps, but also the time spent thinking about the problem
> without the benefit of that information.

ok changed to 'git describe' [and merged with next]

Satish


From bsmith at mcs.anl.gov  Mon Jan  6 21:58:04 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 6 Jan 2014 21:58:04 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401061833150.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
Message-ID: <4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>


On Jan 6, 2014, at 6:36 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 6 Jan 2014, Barry Smith wrote:
> 
>> 
>> On Jan 6, 2014, at 4:52 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>> 
>>> so 'mailto' is no good.
>> 
>>   Not everyone lives in a cave and uses the pony-express to communicate. Are really most PETSc users and developers complete troglodytes?
>> 
> 
> Well - most don't have mailto: configured from their web browser

   Huh? Next you?ll tell me that people don?t have http://?.    configured for their web browsers.  Still using Archie and Veronica for searches I guess.

   Barry



> [eventhough they might use the spiffer new mail clients]
> 
> 
> I guess thats a one time setup cost though..
> 
>>   Note that you can setup mailto links to go to Emacs https://code.google.com/p/plume-lib/source/browse/bin/emacs-mailto-handler?name=v1.0.0 and alpine https://groups.google.com/forum/#!topic/comp.mail.pine/puiLTBBQDBg
>> 
>>   Satish,
>> 
>>   I like the mailto: link you have set up and would like it to stay. It looks great to me but it currently doesn?t do the address correctly: it gives me petsc-maint.anl.gov
>> when it should give petsc-maint at mcs.anl.gov
> 
> fixed.
> 
>> 
>>   Put it into next and master
> 
> ok merged to next for now..
> 
> satish
> 
>> 
>>   Thanks
>> 
>>   Barry
>> 
>>   Nix the form web submission form idea, MCS has such a primitive web server system it will suck.
>> 
>> 
>> 
>>> 
>>> and just adding text "report errors to petsc-maint at mcs.anl.gov"
>>> doesn't improve on status-quo.
>>> 
>>> so 'web submission form' would be the prefered thingy? I'll have to
>>> check if thats feasiable..
>>> 
>>> Satish
>>> 
>>> On Mon, 6 Jan 2014, Nystrom, William D wrote:
>>> 
>>>> But if the user has to submit the error info by emailing manually, would
>>>> you likely lose some of the info you are trying to capture automatically?
>>>> 
>>>> ________________________________________
>>>> From: Jed Brown [jedbrown at mcs.anl.gov]
>>>> Sent: Monday, January 06, 2014 3:25 PM
>>>> To: Aron Ahmadia
>>>> Cc: Nystrom, William D; Satish Balay; petsc-dev
>>>> Subject: Re: [petsc-dev] adding Report Typo and Errors link to manual pages
>>>> 
>>>> Aron Ahmadia <aron at ahmadia.net> writes:
>>>> 
>>>>> Perhaps make the Email Address visible in the URL link?
>>>>> 
>>>>> "Click here to report typos/errors to petsc-maint at mcs.anl.gov".
>>>>> 
>>>>> Make the entire statement a mailto link?  Then anybody who gets in a
>>>>> confusing mailto situation will at least know what the URL was supposed to
>>>>> do (open a bug report).
>>>> 
>>>> Good call, but lose the fluff:
>>>> 
>>>> "report errors to petsc-maint at mcs.anl.gov"
>>>> 
>>> 
>> 
>> 



From jroman at dsic.upv.es  Tue Jan  7 05:33:57 2014
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 7 Jan 2014 12:33:57 +0100
Subject: [petsc-dev] options_left
Message-ID: <3DE9CAC8-CAA2-48E4-949E-64470670DEDB@dsic.upv.es>

Now in 'master' mistyped/unused options are not displayed unless -options_left is specified.
Is this intentional? If so, this change should appear in the list of changes.

Another question: I could use some of the additions to branch prbrune/removeunwrappedmathfunctions, but this branch has not been merged to 'master' yet (the corresponding pull request merged it to 'next'). Is it an oversight?

Jose



From balay at mcs.anl.gov  Tue Jan  7 09:23:53 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 7 Jan 2014 09:23:53 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>,
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401070920430.3067@asterix>

On Mon, 6 Jan 2014, Barry Smith wrote:

> >> 
> >>   Not everyone lives in a cave and uses the pony-express to communicate. Are really most PETSc users and developers complete troglodytes?
> >> 
> > 
> > Well - most don't have mailto: configured from their web browser
> 
>    Huh? Next you?ll tell me that people don?t have http://?.    configured for their web browsers.  Still using Archie and Veronica for searches I guess.
> 

If you are on Mac using Safari for web and Apple Mail for e-mail -
then you are all set.

However - if you use safari for web - and gmail [or any other mail
client] for e-mail - then you have to go through extra steps.

Satish

From knepley at gmail.com  Tue Jan  7 09:24:55 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 7 Jan 2014 09:24:55 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401070920430.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
Message-ID: <CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>

On Tue, Jan 7, 2014 at 9:23 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 6 Jan 2014, Barry Smith wrote:
>
> > >>
> > >>   Not everyone lives in a cave and uses the pony-express to
> communicate. Are really most PETSc users and developers complete
> troglodytes?
> > >>
> > >
> > > Well - most don't have mailto: configured from their web browser
> >
> >    Huh? Next you?ll tell me that people don?t have http://?.
>  configured for their web browsers.  Still using Archie and Veronica for
> searches I guess.
> >
>
> If you are on Mac using Safari for web and Apple Mail for e-mail -
> then you are all set.
>
> However - if you use safari for web - and gmail [or any other mail
> client] for e-mail - then you have to go through extra steps.


Or you can use Chrome and Stop the Madness.

   Matt


>
> Satish




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140107/1f17312f/attachment.html>

From balay at mcs.anl.gov  Tue Jan  7 09:34:58 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 7 Jan 2014 09:34:58 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401070927540.3067@asterix>

On Tue, 7 Jan 2014, Matthew Knepley wrote:

> On Tue, Jan 7, 2014 at 9:23 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Mon, 6 Jan 2014, Barry Smith wrote:
> >
> > > >>
> > > >>   Not everyone lives in a cave and uses the pony-express to
> > communicate. Are really most PETSc users and developers complete
> > troglodytes?
> > > >>
> > > >
> > > > Well - most don't have mailto: configured from their web browser
> > >
> > >    Huh? Next you?ll tell me that people don?t have http://?.
> >  configured for their web browsers.  Still using Archie and Veronica for
> > searches I guess.
> > >
> >
> > If you are on Mac using Safari for web and Apple Mail for e-mail -
> > then you are all set.
> >
> > However - if you use safari for web - and gmail [or any other mail
> > client] for e-mail - then you have to go through extra steps.
> 
> 
> Or you can use Chrome and Stop the Madness.

chrome also defaults to the 'default mailer' for your desktop.

i.e on most linuxes - 'evolution' is set as the default mailer for
gnome [perhaps kmail for kde etc.]. Unless you change this setting to
something else - chrome will attempt to spwan evolution/kmail.

Since 'mailto' requires extra config for setting users prefered mail
client - I assert most users don't have this configured.
[perhaps its a trival config for most common e-mail clients]

Folks on this list can easily give feedback and tell us if thier
browser is configured already for mailto: for their prefrered e-mail
client or not.

http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html

To start off - mine isn't. [yes I live in a cave]

Satish

From knepley at gmail.com  Tue Jan  7 09:41:26 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 7 Jan 2014 09:41:26 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401070927540.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
Message-ID: <CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>

On Tue, Jan 7, 2014 at 9:34 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 7 Jan 2014, Matthew Knepley wrote:
>
> > On Tue, Jan 7, 2014 at 9:23 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > On Mon, 6 Jan 2014, Barry Smith wrote:
> > >
> > > > >>
> > > > >>   Not everyone lives in a cave and uses the pony-express to
> > > communicate. Are really most PETSc users and developers complete
> > > troglodytes?
> > > > >>
> > > > >
> > > > > Well - most don't have mailto: configured from their web browser
> > > >
> > > >    Huh? Next you?ll tell me that people don?t have http://?.
> > >  configured for their web browsers.  Still using Archie and Veronica
> for
> > > searches I guess.
> > > >
> > >
> > > If you are on Mac using Safari for web and Apple Mail for e-mail -
> > > then you are all set.
> > >
> > > However - if you use safari for web - and gmail [or any other mail
> > > client] for e-mail - then you have to go through extra steps.
> >
> >
> > Or you can use Chrome and Stop the Madness.
>
> chrome also defaults to the 'default mailer' for your desktop.
>

? I just tried it and Chrome uses GMail. I have not configured anything.

   Matt


> i.e on most linuxes - 'evolution' is set as the default mailer for
> gnome [perhaps kmail for kde etc.]. Unless you change this setting to
> something else - chrome will attempt to spwan evolution/kmail.
>
> Since 'mailto' requires extra config for setting users prefered mail
> client - I assert most users don't have this configured.
> [perhaps its a trival config for most common e-mail clients]
>
> Folks on this list can easily give feedback and tell us if thier
> browser is configured already for mailto: for their prefrered e-mail
> client or not.
>
> http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html
>
> To start off - mine isn't. [yes I live in a cave]
>
> Satish
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140107/37be9ae3/attachment.html>

From jedbrown at mcs.anl.gov  Tue Jan  7 09:42:20 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 07 Jan 2014 09:42:20 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401070927540.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
Message-ID: <87mwj7g5nn.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html

This is still petsc-maint.anl.gov, should be petsc-maint at mcs.anl.gov.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140107/4c7152bf/attachment.sig>

From jedbrown at mcs.anl.gov  Tue Jan  7 09:46:00 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 07 Jan 2014 09:46:00 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<7374735A-DFBF-4C84-AEB3-3B3D8CED9366@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
	<CAMYG4G=D6_oZhGm-F2zZwXtm JXkEWd6_
	Dg26sjkaMdrezHeDWw@mail.gmail.com>
Message-ID: <87k3ebg5hj.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> ? I just tried it and Chrome uses GMail. I have not configured anything.

I tried with a sandbox account on my Linux system and firefox, chromium,
and google-chrome all go to the OS email configuration dialog,
defaulting to Evolution.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140107/61a6d30a/attachment.sig>

From balay at mcs.anl.gov  Tue Jan  7 09:46:36 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 7 Jan 2014 09:46:36 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
	<CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401070944180.3067@asterix>

On Tue, 7 Jan 2014, Matthew Knepley wrote:

> > > Or you can use Chrome and Stop the Madness.
> >
> > chrome also defaults to the 'default mailer' for your desktop.
> >
> 
> ? I just tried it and Chrome uses GMail. I have not configured anything.
> 

well it spawns evolution for me.

Satish


From balay at mcs.anl.gov  Tue Jan  7 09:51:51 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 7 Jan 2014 09:51:51 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <87mwj7g5nn.fsf@jedbrown.org>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<alpine.LFD.2.11.1401061452300.17785@asterix>
	<87ob3oizz2.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
	<87mwj7g5nn.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401070948440.3067@asterix>

On Tue, 7 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> > http://ftp.mcs.anl.gov/pub/petsc/tmp/VecView.html
> 
> This is still petsc-maint.anl.gov, should be petsc-maint at mcs.anl.gov.

Sorry - that temp page wasn't updated earlier. Its updated now.
[with the version from petsc-next nightly tarball]

Satish



From balay at mcs.anl.gov  Tue Jan  7 09:58:47 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 7 Jan 2014 09:58:47 -0600
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401070944180.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>
	<alpine.LFD.2.11.1401061508310.17785@asterix>
	<87iotwizch.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401061548270.17785@asterix>
	<878uusixg1.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>
	<8761pwiwoq.fsf@jedbrown.org>
	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>
	<87zjn8hhne.fsf@jedbrown.org>
	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>
	<alpine.LFD.2.11.1401061649460.3067@asterix>
	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>
	<alpine.LFD.2.11.1401061833150.3067@asterix>
	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>
	<alpine.LFD.2.11.1401070920430.3067@asterix>
	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>
	<alpine.LFD.2.11.1401070927540.3067@asterix>
	<CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>
	<alpine.LFD.2.11.1401070944180.3067@asterix>
Message-ID: <alpine.LFD.2.11.1401070956370.3067@asterix>

On Tue, 7 Jan 2014, Satish Balay wrote:

> On Tue, 7 Jan 2014, Matthew Knepley wrote:
> 
> > > > Or you can use Chrome and Stop the Madness.
> > >
> > > chrome also defaults to the 'default mailer' for your desktop.
> > >
> > 
> > ? I just tried it and Chrome uses GMail. I have not configured anything.
> > 
> 
> well it spawns evolution for me.

http://googlesystem.blogspot.com/2012/02/open-mailto-links-using-gmail-in-google.html

Looks like if you log into gmail from chrome - it will prompt you to
set gmail as your default 'mailto:' client.

[you might have said 'yes' at some point]

Satish


From bsmith at mcs.anl.gov  Tue Jan  7 11:57:05 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 7 Jan 2014 11:57:05 -0600
Subject: [petsc-dev] options_left
In-Reply-To: <3DE9CAC8-CAA2-48E4-949E-64470670DEDB@dsic.upv.es>
References: <3DE9CAC8-CAA2-48E4-949E-64470670DEDB@dsic.upv.es>
Message-ID: <A3651EB3-4960-4D34-8B7D-2BFFD8B0C023@mcs.anl.gov>


   I accidentally damaged that code in 28176ea I have fixed it in master and next

    Thanks for reporting the problem,

    Barry

On Jan 7, 2014, at 5:33 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> Now in 'master' mistyped/unused options are not displayed unless -options_left is specified.
> Is this intentional? If so, this change should appear in the list of changes.
> 
> Another question: I could use some of the additions to branch prbrune/removeunwrappedmathfunctions, but this branch has not been merged to 'master' yet (the corresponding pull request merged it to 'next'). Is it an oversight?
> 
> Jose
> 



From rupp at mcs.anl.gov  Wed Jan  8 07:34:48 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 08 Jan 2014 14:34:48 +0100
Subject: [petsc-dev] adding Report Typo and Errors link to manual pages
In-Reply-To: <alpine.LFD.2.11.1401070956370.3067@asterix>
References: <70A2B464-1EFD-46C7-BA45-F6BC971B2001@mcs.anl.gov>	<alpine.LFD.2.11.1401061508310.17785@asterix>	<87iotwizch.fsf@jedbrown.org>	<alpine.LFD.2.11.1401061548270.17785@asterix>	<878uusixg1.fsf@jedbrown.org>	<CB90CF2315BF6A44986ACD4A47418D226F151D75@ECS-EXG-P-MB05.win.lanl.gov>	<8761pwiwoq.fsf@jedbrown.org>	<CAPhiW4jeV7UOiJgY-xCyB6r_KG4BN0a2t6cSuL4jjTm6ne=0ig@mail.gmail.com>	<87zjn8hhne.fsf@jedbrown.org>	<CB90CF2315BF6A44986ACD4A47418D226F151DD2@ECS-EXG-P-MB05.win.lanl.gov>	<alpine.LFD.2.11.1401061649460.3067@asterix>	<2BF8B83C-92FD-4EAB-B5D6-B06C5F4807A6@mcs.anl.gov>	<alpine.LFD.2.11.1401061833150.3067@asterix>	<4BEAB64C-DFF9-4281-BB41-08AD18F4D40E@mcs.anl.gov>	<alpine.LFD.2.11.1401070920430.3067@asterix>	<CAMYG4GkW9_kYLw6jzTCAQGa1SraGv_Kq1CPxQx_Tgxy=U2Pmbw@mail.gmail.com>	<alpine.LFD.2.11.1401070927540.3067@asterix>	<CAMYG4G=D6_oZhGm-F2zZwXtmJXkEWd6_Dg26sjkaMdrezHeDWw@mail.gmail.com>	<alpine.LFD.2.11.1401070944180.3067@asterix>
	<alpine.LFD.2.11.1401070956370.3067@ast erix>
Message-ID: <52CD53F8.3070308@mcs.anl.gov>

Hi guys,

 >>>> chrome also defaults to the 'default mailer' for your desktop.
>>>>
>>>
>>> ? I just tried it and Chrome uses GMail. I have not configured anything.
>>>
>>
>> well it spawns evolution for me.
>
> http://googlesystem.blogspot.com/2012/02/open-mailto-links-using-gmail-in-google.html
>
> Looks like if you log into gmail from chrome - it will prompt you to
> set gmail as your default 'mailto:' client.
>
> [you might have said 'yes' at some point]

I live with Satish in a cave and haven't set up the mail client in the 
browser. Winter gives us quite a hard time with the cold out there ;-)

I see the following options:

  a) Keep the mailto: link and assume users will handle it

  b) Let the link forward to another webpage which provides the form. 
This can be any webserver, so we are not restricted by the capabilities 
of the MCS server. The link represents either a call to JavaScript's 
window.open() ('pop-up window'), or a vanilla HTML link opening the form 
in a new tab.

  c) Include some Web 2.0 magic such that a box pops up after clicking 
the link, with the user staying on the page in order to make the 
description of the typo as simple as possible. The disadvantage is that 
a couple of additional HTML lines need to be added to *all* man pages 
even if we place all the JavaScript into a separate (common) file. It 
makes things easier if the MCS server can send emails, but it's not 
required.

b) and c) certainly require some minor spam prevention techniques (e.g. 
Captcha or "Enter 'nospam' here")

Satish, is the MCS webserver capable of sending emails? If not, do you 
know of any good server/webspace we can use to sending such generated 
emails to petsc-maint?

Best regards,
Karli




From jedbrown at mcs.anl.gov  Wed Jan  8 09:43:49 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 08 Jan 2014 08:43:49 -0700
Subject: [petsc-dev] [Andy Ray Terrel] Re: [dev@libelemental.org]
	Vagrantfiles
Message-ID: <87ha9eiimi.fsf@jedbrown.org>

Is this worth doing for PETSc?

-------------- next part --------------
An embedded message was scrubbed...
From: Andy Ray Terrel <andy.terrel at gmail.com>
Subject: Re: [dev at libelemental.org] Vagrantfiles
Date: Tue, 7 Jan 2014 18:10:21 -0600
Size: 7762
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140108/20c37d57/attachment.mht>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140108/20c37d57/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan  8 14:25:38 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 8 Jan 2014 14:25:38 -0600
Subject: [petsc-dev] [Andy Ray Terrel] Re: [dev@libelemental.org]
	Vagrantfiles
In-Reply-To: <87ha9eiimi.fsf@jedbrown.org>
References: <87ha9eiimi.fsf@jedbrown.org>
Message-ID: <F1CAC51A-229E-4CCC-AE3D-F68B9D19A87F@mcs.anl.gov>

An embedded message was scrubbed...
From: Barry Smith <bsmith at mcs.anl.gov>
Subject: PETSc server
Date: Sat, 27 Apr 2013 22:32:44 -0500
Size: 936
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140108/b30b25a6/attachment.eml>
-------------- next part --------------

On Jan 8, 2014, at 9:43 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Is this worth doing for PETSc?
> 
> 
> Resent-From: andy.terrel at gmail.com
> From: Andy Ray Terrel <andy.terrel at gmail.com>
> Subject: Re: [dev at libelemental.org] Vagrantfiles
> Date: January 7, 2014 at 6:10:21 PM CST
> To: dev at libelemental.org
> 
> 
> Yes! Vagrant is the new answer to, "How do I stall this on Windows?"
> It really is as simple as typing vagrant up and then vagrant ssh.
> 
> On Tue, Jan 7, 2014 at 4:51 PM, Jack Poulson <jpoulson at cc.gatech.edu> wrote:
>> Dear Elemental Developers,
>> 
>> I have finally taken the time to learn how to use Vagrant
>> [http://vagrantup.com] and have committed Vagrantfiles for developing
>> Elemental in (headless) 32-bit and 64-bit Ubuntu 13.10:
>> https://github.com/elemental/Elemental/commit/c809bdc9a240f6bdd5403f7c5e278d75415af66d
>> 
>> For those of you that aren't familiar with Vagrant, it is a means of
>> easily maintaining development environments for virtual machines. Once
>> Vagrant is installed, a full development environment for Elemental can
>> be started by simply typing 'vagrant up'.
>> 
>> I hope to add desktop versions, saucy32-desktop and saucy64-desktop,
>> soon so that a dev environment including Qt5 support can easily be
>> loaded. This would be the ideal setting for a quickstart tutorial. (Any
>> takers on helping write such a tutorial? It would obviously be one of
>> the most public-facing pieces of the project.) The complication is that
>> vagrant boxes need to be manually created (I'm in the process of
>> creating one for saucy32-desktop).
>> 
>> Jack
> 
> 


From jedbrown at mcs.anl.gov  Wed Jan  8 14:43:05 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 08 Jan 2014 13:43:05 -0700
Subject: [petsc-dev] [Andy Ray Terrel] Re: [dev@libelemental.org]
	Vagrantfiles
In-Reply-To: <F1CAC51A-229E-4CCC-AE3D-F68B9D19A87F@mcs.anl.gov>
References: <87ha9eiimi.fsf@jedbrown.org>
	<F1CAC51A-229E-4CCC-AE3D-F68B9D19A87F@mcs.anl.gov>
Message-ID: <87a9f6gq7a.fsf@jedbrown.org>

I know we chatted about it, but Vagrant does look easy to use and may be
lighter weight than what Satish tried.  I don't personally like using
VMs, but it would probably help some people.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140108/6ee134ca/attachment.sig>

From balay at mcs.anl.gov  Wed Jan  8 14:46:21 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 8 Jan 2014 14:46:21 -0600
Subject: [petsc-dev] PETSc server
In-Reply-To: <22DD110E-1A22-4832-9B72-F4A0E8A951F0@mcs.anl.gov>
References: <22DD110E-1A22-4832-9B72-F4A0E8A951F0@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401081438420.3067@asterix>

[this multipart mesaage has a funny format and confusing my mailer - so replying thus..]

On Sat, 27 Apr 2013, Barry Smith wrote:

> 
>    Satish,
> 
>     Since installing PETSc is so difficult how hard would it be 

Its usually difficult for user specific settings - and VM/server doesn't adress this primary issue.

> 
> 1) provide an entire linux image with PETSc installed and user account already there that people can run in a VM on their machine?

A simple install on windows can be done via cygwin [with gcc/gfortran
etc.]. A VM will tradeoff installing cygwin with a
installing/managing/VM+linux

The last I checked - a linux VM install with all the required
developer tools [and a useable minimal X-desktop] takes 2G or more.

[I don't think its a big deal for us to host such a big download - but
users would have to deal with the VM issues. Usually these a
compressed with .xz to minimize footprint]

> 
> 2) allow people to log into a VM on a server with PETSc already installed and user account created for them automatically either
>    
>    a) on some machines we maintain or 
>    b) on some Amazon/..../...   server system?

We would be in account-management game instead here. Jed was
previously proposing having the above image available on amazon - and
users would just run instances of it the way they need it..

Satish


From balay at mcs.anl.gov  Wed Jan  8 14:56:04 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 8 Jan 2014 14:56:04 -0600
Subject: [petsc-dev] [Andy Ray Terrel] Re: [dev@libelemental.org]
 Vagrantfiles
In-Reply-To: <87a9f6gq7a.fsf@jedbrown.org>
References: <87ha9eiimi.fsf@jedbrown.org>
	<F1CAC51A-229E-4CCC-AE3D-F68B9D19A87F@mcs.anl.gov>
	<87a9f6gq7a.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401081453550.3067@asterix>

On Wed, 8 Jan 2014, Jed Brown wrote:

> I know we chatted about it, but Vagrant does look easy to use and may be
> lighter weight than what Satish tried.  I don't personally like using
> VMs, but it would probably help some people.

Does 'Vagrant' run images remotely - and you get to ssh to it via vagrant local client?

Satish


From jedbrown at mcs.anl.gov  Wed Jan  8 15:06:53 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 08 Jan 2014 14:06:53 -0700
Subject: [petsc-dev] PETSc server
In-Reply-To: <alpine.LFD.2.11.1401081438420.3067@asterix>
References: <22DD110E-1A22-4832-9B72-F4A0E8A951F0@mcs.anl.gov>
	<alpine.LFD.2.11.1401081438420.3067@asterix>
Message-ID: <877gaagp3m.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> A simple install on windows can be done via cygwin [with gcc/gfortran
> etc.]. A VM will tradeoff installing cygwin with a
> installing/managing/VM+linux

Installing cygwin is more than one command.

> The last I checked - a linux VM install with all the required
> developer tools [and a useable minimal X-desktop] takes 2G or more.

Vagrant wouldn't use a monolithic image.  We would just distribute the
Vagrantfile.  I don't know what the total download size would work out
to, but we wouldn't be the ones hosting it.

> We would be in account-management game instead here. Jed was
> previously proposing having the above image available on amazon - and
> users would just run instances of it the way they need it..

Amazon will host machine images.  Users create an instance and choose
the machine image.  It's pretty easy if you want to use their hardware.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140108/8800bcd3/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan  8 15:56:47 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 8 Jan 2014 15:56:47 -0600
Subject: [petsc-dev] [Andy Ray Terrel] Re: [dev@libelemental.org]
	Vagrantfiles
In-Reply-To: <alpine.LFD.2.11.1401081453550.3067@asterix>
References: <87ha9eiimi.fsf@jedbrown.org>
	<F1CAC51A-229E-4CCC-AE3D-F68B9D19A87F@mcs.anl.gov>
	<87a9f6gq7a.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401081453550.3067@asterix>
Message-ID: <8A844B1B-8708-4CCA-85A2-696177BFF055@mcs.anl.gov>


On Jan 8, 2014, at 2:56 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Wed, 8 Jan 2014, Jed Brown wrote:
> 
>> I know we chatted about it, but Vagrant does look easy to use and may be
>> lighter weight than what Satish tried.  I don't personally like using
>> VMs, but it would probably help some people.
> 
> Does 'Vagrant' run images remotely - and you get to ssh to it via vagrant local client?

   I took a quick look at the docs. It seems to run the VM locally but you ssh to get into it as opposed to it taking over the entire machine maybe?  There is a download button on the front page, try it out, see what happens.

   Barry

> 
> Satish



From andy.terrel at gmail.com  Wed Jan  8 15:56:13 2014
From: andy.terrel at gmail.com (Andy Ray Terrel)
Date: Wed, 8 Jan 2014 15:56:13 -0600
Subject: [petsc-dev] PETSc server
In-Reply-To: <877gaagp3m.fsf@jedbrown.org>
References: <22DD110E-1A22-4832-9B72-F4A0E8A951F0@mcs.anl.gov>
	<alpine.LFD.2.11.1401081438420.3067@asterix>
	<877gaagp3m.fsf@jedbrown.org>
Message-ID: <CA+WonSTzrsUwFHL-nRi3nMnmmzq3bSfH7KoF1M8t5XnRW_UxRw@mail.gmail.com>

I highly recommend Vagrant.  I can set something up for you this week.

You add two files:  Vagrantfile and bootstrap.sh

Then the user only needs to do:

$ cd petsc-dev
$ vagrant up
-- Downloads minimal image and calls the bootstrap to install all dependencies
$ vagrant ssh

Even editting the files is synced so you can use whatever crazy editor
you like on your system.

The best part is you can just install the vagrant-aws or
vagrant-digitalocean plugins and then the same commands will put you
on a server in the cloud.

-- Andy

On Wed, Jan 8, 2014 at 3:06 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Satish Balay <balay at mcs.anl.gov> writes:
>
>> A simple install on windows can be done via cygwin [with gcc/gfortran
>> etc.]. A VM will tradeoff installing cygwin with a
>> installing/managing/VM+linux
>
> Installing cygwin is more than one command.
>
>> The last I checked - a linux VM install with all the required
>> developer tools [and a useable minimal X-desktop] takes 2G or more.
>
> Vagrant wouldn't use a monolithic image.  We would just distribute the
> Vagrantfile.  I don't know what the total download size would work out
> to, but we wouldn't be the ones hosting it.
>
>> We would be in account-management game instead here. Jed was
>> previously proposing having the above image available on amazon - and
>> users would just run instances of it the way they need it..
>
> Amazon will host machine images.  Users create an instance and choose
> the machine image.  It's pretty easy if you want to use their hardware.


From jedbrown at mcs.anl.gov  Thu Jan  9 15:00:46 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 09 Jan 2014 14:00:46 -0700
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
Message-ID: <877ga8dg5d.fsf@jedbrown.org>

"Garth N. Wells" <gnw20 at cam.ac.uk> writes:

> Hi Jed,
>
> I've been poking around the PETSc code to see how objects manage the MPI 
> communicator that is passed in, and running some simple tests using 
> MPI_Comm_compare. Does a PETSc object (say a Vec) always duplicate the 
> communicator? From tests with MPI_Comm_compare this looks to be the 
> case.

The first time a communicator is encountered by a PETSc object,
PetscCommDuplicate dups it, yielding an "inner" communicator.  It
attaches the inner communicator, a reference count, and a tag dispenser
to the outer communicator that you just passed in.  We always use the
tag dispenser to get new tags, always perform operations on the inner
communicator.  The user never handles the inner communicator.
Subsequent calls to PetscCommDuplicate will find the attribute and just
pull out the inner communicator.

The outer communicator can be passed around, including through other
libraries, and back into PETSc, yet there will still only ever be one
inner comm per outer comm.  When an object is destroyed, the reference
count decreases and the inner communicator is destroyed when the
reference count drops to zero.

Barry, this code has been around for ages, but we should write a short
report explaining what we do and why.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140109/0971463d/attachment.sig>

From bsmith at mcs.anl.gov  Thu Jan  9 15:54:11 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 9 Jan 2014 15:54:11 -0600
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <877ga8dg5d.fsf@jedbrown.org>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
Message-ID: <C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>


On Jan 9, 2014, at 3:00 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> "Garth N. Wells" <gnw20 at cam.ac.uk> writes:
> 
>> Hi Jed,
>> 
>> I've been poking around the PETSc code to see how objects manage the MPI 
>> communicator that is passed in, and running some simple tests using 
>> MPI_Comm_compare. Does a PETSc object (say a Vec) always duplicate the 
>> communicator? From tests with MPI_Comm_compare this looks to be the 
>> case.
> 
> The first time a communicator is encountered by a PETSc object,
> PetscCommDuplicate dups it, yielding an "inner" communicator.  It
> attaches the inner communicator, a reference count, and a tag dispenser
> to the outer communicator that you just passed in.  We always use the
> tag dispenser to get new tags, always perform operations on the inner
> communicator.  The user never handles the inner communicator.
> Subsequent calls to PetscCommDuplicate will find the attribute and just
> pull out the inner communicator.
> 
> The outer communicator can be passed around, including through other
> libraries, and back into PETSc, yet there will still only ever be one
> inner comm per outer comm.  When an object is destroyed, the reference
> count decreases and the inner communicator is destroyed when the
> reference count drops to zero.
> 
> Barry, this code has been around for ages, but we should write a short
> report explaining what we do and why.

   We do this because it is necessary to do this (or something similar), for any MPI based library. I am hoping that MPI will be replaced by another programming model before it is necessary to get the wider community to understand why it is necessary to do this and how we do it :-(

   Perhaps we should pull out the code into its own little library that can be used by the various library writing groups? 

   There are many things we could/should do, but how much time to we want to spend on dying legacies :-)


  Barry






From jedbrown at mcs.anl.gov  Thu Jan  9 16:03:10 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 09 Jan 2014 15:03:10 -0700
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
Message-ID: <87ppo0byox.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    We do this because it is necessary to do this (or something
>    similar), for any MPI based library. I am hoping that MPI will be
>    replaced by another programming model before it is necessary to get
>    the wider community to understand why it is necessary to do this
>    and how we do it :-(

It has been 20 years and most libraries still create wrapper classes of
some sort, which are much worse.

>    Perhaps we should pull out the code into its own little library
>    that can be used by the various library writing groups?

That might make sense.  Probably nobody wants to depend on such a thing
as an external library, but they could copy a single source file into
their tree (or git subtree merge it in).

>    There are many things we could/should do, but how much time to we
>    want to spend on dying legacies :-)

I'll plan for an MPI replacement when people designing such systems take
the effort to understand MPI.  So far, they are building sand castles
and talking about how metals and composites won't scale as construction
and aerospace materials.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140109/bba71e62/attachment.sig>

From bsmith at mcs.anl.gov  Thu Jan  9 16:38:21 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 9 Jan 2014 16:38:21 -0600
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <87ppo0byox.fsf@jedbrown.org>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
	<87ppo0byox.fsf@jedbrown.org>
Message-ID: <6E253A59-ED30-4191-A668-9B7377548848@mcs.anl.gov>


On Jan 9, 2014, at 4:03 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>> 
> 
> I'll plan for an MPI replacement when people designing such systems take
> the effort to understand MPI.

     This is a fundamental problem and has been a problem since soon after MPI was created. People think they can design new HPC programming models without having more than an incredibly superficial of the one successful HPC programming model. 



From karpeev at mcs.anl.gov  Thu Jan  9 16:42:05 2014
From: karpeev at mcs.anl.gov (Dmitry Karpeyev)
Date: Thu, 9 Jan 2014 16:42:05 -0600
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <b9c77088c01a47c6b635869afb21ade3@NAGURSKI.anl.gov>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
	<87ppo0byox.fsf@jedbrown.org>
	<b9c77088c01a47c6b635869afb21ade3@NAGURSKI.anl.gov>
Message-ID: <CA+Y_9UJH64FeyAhcSsArYjOEnSYSA0JUUP-R=vC232A-OBfC_w@mail.gmail.com>

I'd say that a writeup explaining why the inner comms are needed would be
useful also for others to understand the peculiarities/shortcomings of the
MPI model.  You never know who might be contributing to the next successful
programming model, so don't we want more people to understand MPI?

Dmitry.


On Thu, Jan 9, 2014 at 4:38 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
> On Jan 9, 2014, at 4:03 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
> > Barry Smith <bsmith at mcs.anl.gov> writes:
> >>
> >
> > I'll plan for an MPI replacement when people designing such systems take
> > the effort to understand MPI.
>
>      This is a fundamental problem and has been a problem since soon after
> MPI was created. People think they can design new HPC programming models
> without having more than an incredibly superficial of the one successful
> HPC programming model.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140109/bff9c1e7/attachment.html>

From jedbrown at mcs.anl.gov  Thu Jan  9 16:56:21 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 09 Jan 2014 15:56:21 -0700
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <CA+Y_9UJH64FeyAhcSsArYjOEnSYSA0JUUP-R=vC232A-OBfC_w@mail.gmail.com>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
	<87ppo0byox.fsf@jedbrown.org>
	<b9c77088c01a47c6b635869afb21ade3@NAGURSKI.anl.gov>
	<CA+Y_9UJH64FeyAhcSsArYjOEnSYSA0JUUP-R=vC232A-OBfC_w@mail.gmail.com>
Message-ID: <87bnzkbw8a.fsf@jedbrown.org>

Dmitry Karpeyev <karpeev at mcs.anl.gov> writes:

> I'd say that a writeup explaining why the inner comms are needed would be
> useful also for others to understand the peculiarities/shortcomings of the
> MPI model.  

I don't think this is a shortcoming; MPI attributes are perfectly
sufficient for this purpose.  I think Bill had this use case in mind
when arguing for attribute caching in MPI-1.

> You never know who might be contributing to the next successful
> programming model, so don't we want more people to understand MPI?

Yes, that's why I suggested writing a short report explaining Barry and
Bill's reasoning from 20 years ago.  A blog post would be an alternative
medium.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140109/350d6cb4/attachment.sig>

From gnw20 at cam.ac.uk  Thu Jan  9 17:01:14 2014
From: gnw20 at cam.ac.uk (Garth N. Wells)
Date: Thu, 9 Jan 2014 23:01:14 +0000
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <87bnzkbw8a.fsf@jedbrown.org>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
	<87ppo0byox.fsf@jedbrown.org>
	<b9c77088c01a47c6b635869afb21ade3@NAGURSKI.anl.gov>
	<CA+Y_9UJH64FeyAhcSsArYjOEnSYSA0JUUP-R=vC232A-OBfC_w@mail.gmail.com>
	<87bnzkbw8a.fsf@jedbrown.org>
Message-ID: <94813B3F-6418-4B33-A47C-43AB4A3DFCF1@cam.ac.uk>


On 9 Jan 2014, at 22:56, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Dmitry Karpeyev <karpeev at mcs.anl.gov> writes:
> 
>> I'd say that a writeup explaining why the inner comms are needed would be
>> useful also for others to understand the peculiarities/shortcomings of the
>> MPI model.  
> 
> I don't think this is a shortcoming; MPI attributes are perfectly
> sufficient for this purpose.  I think Bill had this use case in mind
> when arguing for attribute caching in MPI-1.
> 
>> You never know who might be contributing to the next successful
>> programming model, so don't we want more people to understand MPI?
> 
> Yes, that's why I suggested writing a short report explaining Barry and
> Bill's reasoning from 20 years ago.  A blog post would be an alternative
> medium.

Looks like it?s sketched at

    http://www.mcs.anl.gov/research/projects/mpi/tutorial/gropp/node115.html

but some more flesh would be helpful.

Garth

From bsmith at mcs.anl.gov  Thu Jan  9 17:12:55 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 9 Jan 2014 17:12:55 -0600
Subject: [petsc-dev] PETSc object management of MPI communicators
In-Reply-To: <87bnzkbw8a.fsf@jedbrown.org>
References: <5de49f9342b7689aeaca8c9e15daed44@cam.ac.uk>
	<877ga8dg5d.fsf@jedbrown.org>
	<C1059568-3AF1-459E-A5BD-4B0757BA3361@mcs.anl.gov>
	<87ppo0byox.fsf@jedbrown.org>
	<b9c77088c01a47c6b635869afb21ade3@NAGURSKI.anl.gov>
	<CA+Y_9UJH64FeyAhcSsArYjOEnSYSA0JUUP-R=vC232A-OBfC_w@mail.gmail.com>
	<87bnzkbw8a.fsf@jedbrown.org>
Message-ID: <CA245468-E9CF-43B9-B3FD-CA5C9BB17C1A@mcs.anl.gov>


   You overstate my thought processes. I actually started with the most naive approach possible and gradually refined it to the present form as roadblocks to the naiver approaches appeared. Satish also contributed to getting the code right a great deal.   Bill knew how to do it all along, of course, he just didn?t tell us the needed details.

   Barry


On Jan 9, 2014, at 4:56 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Dmitry Karpeyev <karpeev at mcs.anl.gov> writes:
> 
>> I'd say that a writeup explaining why the inner comms are needed would be
>> useful also for others to understand the peculiarities/shortcomings of the
>> MPI model.  
> 
> I don't think this is a shortcoming; MPI attributes are perfectly
> sufficient for this purpose.  I think Bill had this use case in mind
> when arguing for attribute caching in MPI-1.
> 
>> You never know who might be contributing to the next successful
>> programming model, so don't we want more people to understand MPI?
> 
> Yes, that's why I suggested writing a short report explaining Barry and
> Bill's reasoning from 20 years ago.  A blog post would be an alternative
> medium.



From eijkhout at tacc.utexas.edu  Sat Jan 11 19:39:34 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Sun, 12 Jan 2014 01:39:34 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <87wqihtdc8.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
	<87wqihtdc8.fsf@jedbrown.org>
Message-ID: <F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>


On Jan 3, 2014, at 7:16 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Also, you
> can't implement an efficient setup for your communication pattern
> without richer semantics, see PetscCommBuildTwoSided_Ibarrier and the paper
> 
>  http://unixer.de/publications/img/hoefler-dsde-protocols.pdf

So I started reading this and came across the bottom of page 2:

> Not all applications change communication patterns frequently. The static version of the SDE has been analyzed in previous works and possible optimizations are well documented (e.g., [3, 4]). How- ever, the dynamic variant, which is important to many irregular applications, received less attention so far.

Right. "Important to many applications" but no references or even names given.

So, in what important applications do the communication patterns change frequently?

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From jedbrown at mcs.anl.gov  Sun Jan 12 22:26:52 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 12 Jan 2014 23:26:52 -0500
Subject: [petsc-dev] IMP prototype
In-Reply-To: <F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
	<87wqihtdc8.fsf@jedbrown.org>
	<F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>
Message-ID: <878uukecc3.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:

> Right. "Important to many applications" but no references or even names given.

I would have asked 

> So, in what important applications do the communication patterns change frequently?

Self-contact, AMR, immersed boundary, particles with non-uniform density
(e.g., NAMD and other applications using Charm++), some matrix
partitioning methods, operations like matrix-matrix multiplication using
matrices that change somehow, including coarse grids of algebraic
multigrid that are affected by thresholding, to name a few.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140112/672b893a/attachment.sig>

From eijkhout at tacc.utexas.edu  Sun Jan 12 22:40:23 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Mon, 13 Jan 2014 04:40:23 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <878uukecc3.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
	<87wqihtdc8.fsf@jedbrown.org>
	<F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>
	<878uukecc3.fsf@jedbrown.org>
Message-ID: <589B939B-28D3-4292-B90B-F11FEC4219B0@tacc.utexas.edu>


On Jan 12, 2014, at 10:26 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Self-contact, AMR,

Yeah, I can see that.

Ok, so I read the paper. I sounds very much like algorithms I've seen before, except that before MPI 3 you had to spell out the non-blocking barrier explicitly. But that's beside the point. What was your point with this paper? I'll agree that there are applications where the connectivity changes quite regularly, and so you need a Log(P) setup, rather than the normal O(P). But it stays a Bulk Synchronous scheme that is in no way hard to fit in the current IMP design; I just haven't done so yet. So what was your reason for bringing it up?

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From jedbrown at mcs.anl.gov  Mon Jan 13 01:55:06 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 13 Jan 2014 02:55:06 -0500
Subject: [petsc-dev] IMP prototype
In-Reply-To: <589B939B-28D3-4292-B90B-F11FEC4219B0@tacc.utexas.edu>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
	<87wqihtdc8.fsf@jedbrown.org>
	<F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>
	<878uukecc3.fsf@jedbrown.org>
	<589B939B-28D3-4292-B90B-F11FEC4219B0@tacc.utexas.edu>
Message-ID: <8738kse2p1.fsf@jedbrown.org>

Victor Eijkhout <eijkhout at tacc.utexas.edu> writes:
> Ok, so I read the paper. I sounds very much like algorithms I've seen
> before, except that before MPI 3 you had to spell out the non-blocking
> barrier explicitly. But that's beside the point. What was your point
> with this paper? I'll agree that there are applications where the
> connectivity changes quite regularly, and so you need a Log(P) setup,
> rather than the normal O(P). But it stays a Bulk Synchronous scheme
> that is in no way hard to fit in the current IMP design; I just
> haven't done so yet. So what was your reason for bringing it up?

I was asking whether you could *implement* this algorithm using IMP.  It
sounds like you are suggesting extending IMP to include it as a
primitive.  That's okay, but increasing the number of primitives makes
the model more complicated and I think it is generally a sign of
weakness when a base abstraction needs many primitives.  It is typically
not easy for users to extend primitives.  If you just need a few and
then the abstraction is rock solid for every purpose that comes along,
great.  But if the model frequently fails to express the desired method,
it either needs work or serious thought to make it play well with
others.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140113/990a6862/attachment.sig>

From knepley at gmail.com  Mon Jan 13 07:37:58 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 13 Jan 2014 07:37:58 -0600
Subject: [petsc-dev] bug in MatZeroRowsColumns for MPIAIJ
In-Reply-To: <52BEE7C6.40609@imperial.ac.uk>
References: <529A2AB2.9040004@imperial.ac.uk>
	<CAMYG4GniTfYmUKCDJf1UC_SwF6=4C0=Fd9E8QY-rz5cGRY1GJA@mail.gmail.com>
	<52BEE7C6.40609@imperial.ac.uk>
Message-ID: <CAMYG4Gn2e4-PpJ02VyRBvicmYuwFB8JgW+3A_ZqmXE4HQ8UjBA@mail.gmail.com>

On Sat, Dec 28, 2013 at 9:01 AM, Stephan Kramer <s.kramer at imperial.ac.uk>wrote:

> On 19/12/13 21:03, Matthew Knepley wrote:
>
>  On Sat, Nov 30, 2013 at 12:13 PM, Stephan Kramer <s.kramer at imperial.ac.uk<mailto:
>> s.kramer at imperial.ac.uk>> wrote:
>>
>>     Dear petsc-devs
>>
>>     Trying to write a test case for MatZeroRowColumns (as requested by
>> Matt) in order to extend it to MPIBAIJ, I came across a bug in the current
>> implementation for MPIAIJ. The attached test code
>>     assembles a simple 2D Laplacian (with the standard 4,-1,-1,-1
>> stencil) on the following grid:
>>
>>     0 1 2  3
>>     4 5 6  7
>>     8 9 10 11
>>
>>     with boundary conditions applied to nodes 0,1,2,3,4 and 8 with values
>> (stored in Vec x) 0., 1., 2., 3., 4. and 8. For simplicity I've kept a zero
>> rhs (before the call to MatZeroRowsColumns)  After
>>     the call to MatZeroRowColumns I therefore expect the following rhs:
>>
>>     0. 1. 2. 3. 4. 5. 2. 3. 8. 8. 0. 0.
>>
>>     The entries at 0,1,2,3,4 and 8 correspond to the boundary values
>> (multiplied by a diagonal value of 1.0) and the values at 5,6,7 and 9 to
>> the the lifted values: 1.+4.=5.,2.,3. and 8. However, if
>>     you run the attached example that's not what you get, the lifted
>> values at 6 and 7 are wrong. This works fine in serial (you have to change
>> nlocal=4 to get the same grid).
>>
>>     I think this is caused by the way that entries in x are communicated
>> with other processes that have off-diagonal entries referencing those. In
>> line 944 of mpiaij.c it uses a VecScatter with
>>     ADD_VALUES to do this, but it uses the local work vector l->lvec
>> without zeroing it first. This means it will still have values in it of
>> whatever it was used for before. In this case that would be
>>     the MatMult of line 125 of test.c (in fact you can "fix" the example
>> by adding the line 136-139 where it does a spurious MatMult with a zero
>> vector).
>>
>>     The attached code could be easily extended to test block matrices
>> (simply increasing bs). The commented out lines in the assembly were
>> basically to make the problem a bit more challenging (have an
>>     asymmetric stencil and the setting of off-process boundary nodes).
>> The expected results for MPIBAIJ can be obtained by comparing with the
>> MPIAIJ results (once that works correctly).
>>
>>
>> Okay, I have put this in knepley/feature-mat-zerorowscols-baij and
>> pushed to next. I also fixed the bug that you found. It should
>> not take much over the holidays to get the BAIJ functionality you want.
>>
>>    Thanks,
>>
>>       Matt
>>
>
> Great. Let me know if there's anything else I can do to help. Just a small
> heads up: the new src/mat/examples/tests/ex18.c has an uninitialized bool
> nonlocalBC
>

Okay, I have checked in preliminary code. It passes the tests for Mat ex18.
I have merged it to next. Let me know how
it works for you.

  Thanks,

     Matt


> Cheers
> Stephan
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140113/808e2e59/attachment.html>

From eijkhout at tacc.utexas.edu  Mon Jan 13 10:06:15 2014
From: eijkhout at tacc.utexas.edu (Victor Eijkhout)
Date: Mon, 13 Jan 2014 16:06:15 +0000
Subject: [petsc-dev] IMP prototype
In-Reply-To: <8738kse2p1.fsf@jedbrown.org>
References: <409B6C6B-8255-474A-B970-28EA5395D7BB@tacc.utexas.edu>
	<8761q5h0w9.fsf@jedbrown.org>
	<2F3F1351-BC9F-48D2-B199-AEB42A4C2A06@tacc.utexas.edu>
	<87wqikg200.fsf@jedbrown.org>
	<58BB1213-FF28-42CF-9724-9D6081868DCC@tacc.utexas.edu>
	<878uuyweuv.fsf@jedbrown.org>
	<DB81CAC0-187E-474C-867F-0D387962EC78@tacc.utexas.edu>
	<8738l6wcnt.fsf@jedbrown.org>
	<E86592CB-98B0-4C98-9634-F1AECCA150C5@tacc.utexas.edu>
	<871u0pvh2u.fsf@jedbrown.org>
	<A1ED39E3-FA11-4EF8-9297-7D0B8EE862EE@tacc.utexas.edu>
	<87wqihtdc8.fsf@jedbrown.org>
	<F01FAC25-846D-49A6-A23E-094DEE02D8EC@tacc.utexas.edu>
	<878uukecc3.fsf@jedbrown.org>
	<589B939B-28D3-4292-B90B-F11FEC4219B0@tacc.utexas.edu>
	<8738kse2p1.fsf@jedbrown.org>
Message-ID: <C6FE9FF9-E849-4C6D-8EE2-9F9ED3DED9E1@tacc.utexas.edu>


On Jan 13, 2014, at 1:55 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

>  It
> sounds like you are suggesting extending IMP to include it as a
> primitive.

More or less. More less than more.

>  That's okay, but increasing the number of primitives makes
> the model more complicated

I don't look at it that way. The basic abstraction is that of distributions. Then there is that of kernels, which have two distributions associated with them, the alpha and the beta. From those two you get data dependencies, whether you call that dataflow, a DAG, or messages. The Sparse Data Exchange is not part of the model: it's the implementation of how data dependencies get realized in the specific case of message passing. The model only says "there will be a mechanism for finding data dependencies", not how it's implemented.

There is still something like "the number of primitives" because in the MPI case you may have a switch on a low level: "if one of the distributions is the redundant duplication, then use an Allgather, otherwise use the Hoefler Sparse Data Exchange". But I don't consider that inelegant: any system will have some of these runtime decisions.

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin





From mc0710 at gmail.com  Mon Jan 13 21:28:44 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 13 Jan 2014 21:28:44 -0600
Subject: [petsc-dev] SNES ex19 not using GPU despite passing the options
Message-ID: <CAJzU6sBQFB+VWeqo4A_GW9vo0TOgF12FYSCURx7W3EPA+y5jRw@mail.gmail.com>

Hi,

I tried to run SNES ex19 with the following options but it does not seem to
use my GPU. See the log summary attached. Am I interpretting the log
summary wrong? I don't see any CUSP calls to copy data from the CPU to the
GPU.

/ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none
-dmmg_nlevels 1 -da_grid_x 300 -da_grid_y 300 -log_summary -mat_no_inode
-preload off  -cusp_synchronize -cuda_s
et_device 0

I get the following output when I do -cuda_show_devices
CUDA device 0: Quadro FX 1800M

Cheers,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140113/c7c819bb/attachment.html>
-------------- next part --------------
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex19 on a arch-linux2-c-debug named Deathstar with 1 processor, by mc Mon Jan 13 18:23:03 2014
Using Petsc Development GIT revision: v3.4.3-2308-gdf38795  GIT Date: 2014-01-13 15:21:37 -0600

                         Max       Max/Min        Avg      Total
Time (sec):           1.711e+02      1.00000   1.711e+02
Objects:              9.300e+01      1.00000   9.300e+01
Flops:                3.868e+11      1.00000   3.868e+11  3.868e+11
Flops/sec:            2.260e+09      1.00000   2.260e+09  2.260e+09
MPI Messages:         0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00      0.00000   0.000e+00  0.000e+00
MPI Reductions:       1.250e+02      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total
 0:      Main Stage: 1.7111e+02 100.0%  3.8676e+11 100.0%  0.000e+00   0.0%  0.000e+00        0.0%  1.240e+02  99.2%

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

SNESSolve              1 1.0 1.7075e+02 1.0 3.87e+11 1.0 0.0e+00 0.0e+00 1.0e+02100100  0  0 81 100100  0  0 81  2265
SNESFunctionEval       1 1.0 4.6270e-03 1.0 7.56e+06 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  2   0  0  0  0  2  1634
SNESJacobianEval       1 1.0 4.2646e-01 1.0 1.74e+08 1.0 0.0e+00 0.0e+00 3.1e+01  0  0  0  0 25   0  0  0  0 25   408
VecMDot            10000 1.0 3.3502e+01 1.0 1.12e+11 1.0 0.0e+00 0.0e+00 0.0e+00 20 29  0  0  0  20 29  0  0  0  3329
VecNorm            10335 1.0 3.0934e+00 1.0 7.44e+09 1.0 0.0e+00 0.0e+00 0.0e+00  2  2  0  0  0   2  2  0  0  0  2406
VecScale           10334 1.0 1.7706e+00 1.0 3.72e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  2101
VecCopy            10688 1.0 2.7695e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
VecSet               379 1.0 7.1906e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY              687 1.0 2.3886e-01 1.0 4.95e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2071
VecMAXPY           10334 1.0 2.6452e+01 1.0 1.19e+11 1.0 0.0e+00 0.0e+00 0.0e+00 15 31  0  0  0  15 31  0  0  0  4488
VecScatterBegin       22 1.0 2.0290e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize       10334 1.0 4.8779e+00 1.0 1.12e+10 1.0 0.0e+00 0.0e+00 0.0e+00  3  3  0  0  0   3  3  0  0  0  2288
MatMult            10333 1.0 1.0230e+02 1.0 1.45e+11 1.0 0.0e+00 0.0e+00 0.0e+00 60 37  0  0  0  60 37  0  0  0  1414
MatAssemblyBegin       2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         2 1.0 2.0108e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatZeroEntries         1 1.0 4.2181e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatFDColorCreate       1 1.0 9.2912e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  2   0  0  0  0  2     0
MatFDColorSetUp        1 1.0 2.7306e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.5e+01  0  0  0  0 20   0  0  0  0 20     0
MatFDColorApply        1 1.0 1.4697e-01 1.0 1.74e+08 1.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  2   0  0  0  0  2  1183
MatFDColorFunc        21 1.0 9.5552e-02 1.0 1.59e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1662
KSPGMRESOrthog     10000 1.0 5.8308e+01 1.0 2.23e+11 1.0 0.0e+00 0.0e+00 0.0e+00 34 58  0  0  0  34 58  0  0  0  3825
KSPSetUp               1 1.0 2.3439e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  8   0  0  0  0  8     0
KSPSolve               1 1.0 1.7032e+02 1.0 3.87e+11 1.0 0.0e+00 0.0e+00 6.8e+01100100  0  0 54 100100  0  0 55  2270
PCSetUp                1 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCApply            10334 1.0 2.6863e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

                SNES     1              1         1280     0
      SNESLineSearch     1              1          820     0
              DMSNES     1              1          680     0
              Vector    48             48     64871552     0
      Vector Scatter     3              3         1956     0
              Matrix     1              1     63209092     0
  Matrix FD Coloring     1              1    145133788     0
    Distributed Mesh     1              1       364968     0
     Bipartite Graph     2              2         1632     0
           Index Set    27             27      1820936     0
   IS L to G Mapping     3              3         1788     0
       Krylov Solver     1              1        10164     0
     DMKSP interface     1              1          664     0
      Preconditioner     1              1          808     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 0
#PETSc Option Table entries:
-cuda_set_device 0
-cusp_synchronize
-da_grid_x 300
-da_grid_y 300
-da_mat_type mpiaijcusp
-da_vec_type mpicusp
-dmmg_nlevels 1
-log_summary
-mat_no_inode
-pc_type none
-preload off
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with single precision PetscScalar and PetscReal
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 4 sizeof(PetscInt) 4
Configure run at: Mon Jan 13 17:26:35 2014
Configure options: --prefix=/home/mc/Downloads/petsc_float_optimized/ --with-clean=1 --with-precision=single --with-cuda=1 --with-cuda-only=0 --with-cusp=1 --with-thrust=1 --with-opencl=1 --with-debugging=0 COPTFLAGS="-O3 -march=native" --with-cusp-dir=/home/mc/Downloads/cusplibrary --with-cuda-dir=/opt/cuda --download-txpetscgpu=yes
-----------------------------------------
Libraries compiled on Mon Jan 13 17:26:35 2014 on Deathstar
Machine characteristics: Linux-3.12.7-1-ARCH-x86_64-with-glibc2.2.5
Using PETSc directory: /home/mc/Downloads/petsc
Using PETSc arch: arch-linux2-c-debug
-----------------------------------------

Using C compiler: mpicc  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -march=native  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: mpif90  -fPIC -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O   ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------

Using include paths: -I/home/mc/Downloads/petsc/arch-linux2-c-debug/include -I/home/mc/Downloads/petsc/include -I/home/mc/Downloads/petsc/include -I/home/mc/Downloads/petsc/arch-linux2-c-debug/include -I/opt/cuda/include -I/home/mc/Downloads/cusplibrary/ -I/home/mc/Downloads/cusplibrary/include
-----------------------------------------

Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/home/mc/Downloads/petsc/arch-linux2-c-debug/lib -L/home/mc/Downloads/petsc/arch-linux2-c-debug/lib -lpetsc -llapack -lblas -lX11 -lpthread -Wl,-rpath,/opt/cuda/lib64 -L/opt/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse -lOpenCL -lm -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -Wl,-rpath,/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -Wl,-rpath,/opt/intel/composerxe/compiler/lib/intel64 -L/opt/intel/composerxe/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composerxe/ipp/lib/intel64 -L/opt/intel/composerxe/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composerxe/mkl/lib/intel64 -L/opt/intel/composerxe/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composerxe/tbb/lib/intel64/gcc4.4 -L/opt/intel/composerxe/tbb/lib/intel64/gcc4.4 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl
-----------------------------------------

From mc0710 at gmail.com  Mon Jan 13 21:28:58 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 13 Jan 2014 21:28:58 -0600
Subject: [petsc-dev] Residual evaluation on GPU using OpenCL
Message-ID: <CAJzU6sBdakOMrab1Sf+39X37OmDfF7UV1LaQn_DbmyrV+CWU4Q@mail.gmail.com>

Hi,

ex42 shows how one can do the residual evaluation on the GPU but it uses
Cusp and Thrust. Is there any way one can use OpenCL to evaluate the
residual on the GPU without copying the data back and forth from the CPU
after each iteration?

Also, what is the difference between -da_vec_type seqcusp and -dm_vec_type
seqcusp?

Cheers,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140113/fd871b54/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 14 03:34:27 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 14 Jan 2014 10:34:27 +0100
Subject: [petsc-dev] SNES ex19 not using GPU despite passing the options
In-Reply-To: <CAJzU6sBQFB+VWeqo4A_GW9vo0TOgF12FYSCURx7W3EPA+y5jRw@mail.gmail.com>
References: <CAJzU6sBQFB+VWeqo4A_GW9vo0TOgF12FYSCURx7W3EPA+y5jRw@mail.gmail.com>
Message-ID: <52D504A3.502@mcs.anl.gov>

Hi Mani,

the options are misspelled, you want to use
-dm_vec_type cusp -dm_mat_type aijcusp
(you can omit the 'mpi' to support both the MPI case and the sequential 
case).

I also suggest doing all development in debug mode, in which case you 
would have received the following hints:

WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
Option left: name:-da_mat_type value: mpiaijcusp
Option left: name:-da_vec_type value: mpicusp
Option left: name:-dmmg_nlevels value: 1
Option left: name:-preload value: off

Best regards,
Karli



On 01/14/2014 04:28 AM, Mani Chandra wrote:
> Hi,
>
> I tried to run SNES ex19 with the following options but it does not seem
> to use my GPU. See the log summary attached. Am I interpretting the log
> summary wrong? I don't see any CUSP calls to copy data from the CPU to
> the GPU.
>
> /ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none
> -dmmg_nlevels 1 -da_grid_x 300 -da_grid_y 300 -log_summary -mat_no_inode
> -preload off  -cusp_synchronize -cuda_s
> et_device 0
>
> I get the following output when I do -cuda_show_devices
> CUDA device 0: Quadro FX 1800M
>
> Cheers,
> Mani



From mfadams at lbl.gov  Tue Jan 14 07:06:45 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 14 Jan 2014 08:06:45 -0500
Subject: [petsc-dev] configuring on Titan
Message-ID: <CADOhEh4aBF=wwQSDGxVVo76J=UFus9C-JSwv6RioJSH7UZZGSQ@mail.gmail.com>

I'm trying to configure on Titan.  It tells me to run the compute node job
but when I run the resulting reconfigure script it tells me to
run conftest-arch-titan-opt again (infinite loop).  So I do:

mv ./conftest-arch-titan-opt $MEMBERWORK/env003/

adams/env003> aprun -n 1 ./conftest-arch-titan-opt

mv $MEMBERWORK/env003/reconfigure-arch-titan-opt.py .
./reconfigure-arch-titan-opt.py
vi ./reconfigure-arch-titan-opt.py
./reconfigure-arch-titan-opt.py

I need to edit reconfigure:

*******************************************************************************
                ERROR in COMMAND LINE ARGUMENT to ./configure
-------------------------------------------------------------------------------
The option --known-mpi-int64_t should probably be --known-mpi-int64-t

I just delete this line.  The correct one seems to be there also.

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/ed41d56d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 3377155 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/ed41d56d/attachment.obj>

From triou at cea.fr  Tue Jan 14 08:42:24 2014
From: triou at cea.fr (Projet_TRIOU)
Date: Tue, 14 Jan 2014 15:42:24 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized build)
Message-ID: <52D54CD0.6070608@cea.fr>

Hi all,

I try running in parallel the ex19 test case on CPU and GPU:

OPT="_opt"
dir=/ccc/cont002/home/den/triou/Version_test_airain-hybrid/Trio_U/lib/src/LIBPETSC/petsc/linux$OPT/src/snes/examples/tutorials
option="-pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol 
1.e-5 -log_summary -ksp_view -cuda_show_devices"
mpirun -np 2 $dir/ex19 $option 1>cpu$OPT.log 2>&1
mpirun -np 2 $dir/ex19 -dm_vec_type cusp -dm_mat_type aijcusp $option 
1>gpu$OPT.log 2>&1

With OPT="", PETSc optimized library is used, parallel calculation runs 
well on CPU and GPU.
With OPT="_opt", PETSc non optimized library is used, parallel 
calculation crashes on GPU (it is OK on CPU).

I join the log files. The only difference seems that PETSc-dev is built 
with -O3 intead of -g...
I could try to rebuild PETSc with -O2 but do you have any idea of the 
problem ?

Thanks,

Pierre



-------------- next part --------------
A non-text attachment was scrubbed...
Name: cpu.log
Type: text/x-log
Size: 15683 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/25b176c2/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cpu_opt.log
Type: text/x-log
Size: 14744 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/25b176c2/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gpu.log
Type: text/x-log
Size: 15467 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/25b176c2/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gpu_opt.log
Type: text/x-log
Size: 7867 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/25b176c2/attachment-0003.bin>

From rupp at mcs.anl.gov  Tue Jan 14 09:41:44 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 14 Jan 2014 16:41:44 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
 build)
In-Reply-To: <52D54CD0.6070608@cea.fr>
References: <52D54CD0.6070608@cea.fr>
Message-ID: <52D55AB8.9060301@mcs.anl.gov>

Hi Pierre,

 > I try running in parallel the ex19 test case on CPU and GPU:
>
> OPT="_opt"
> dir=/ccc/cont002/home/den/triou/Version_test_airain-hybrid/Trio_U/lib/src/LIBPETSC/petsc/linux$OPT/src/snes/examples/tutorials
>
> option="-pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol
> 1.e-5 -log_summary -ksp_view -cuda_show_devices"
> mpirun -np 2 $dir/ex19 $option 1>cpu$OPT.log 2>&1
> mpirun -np 2 $dir/ex19 -dm_vec_type cusp -dm_mat_type aijcusp $option
> 1>gpu$OPT.log 2>&1
>
> With OPT="", PETSc optimized library is used, parallel calculation runs
> well on CPU and GPU.
> With OPT="_opt", PETSc non optimized library is used, parallel
> calculation crashes on GPU (it is OK on CPU).
>
> I join the log files. The only difference seems that PETSc-dev is built
> with -O3 intead of -g...
> I could try to rebuild PETSc with -O2 but do you have any idea of the
> problem ?

I could reproduce the problem and also get some uninitialized variable 
warnings in Valgrind. The debug version detects these errors, hence you 
only see the errors in the debug build. For the optimized build, chances 
are good that the computed values are either wrong or may become wrong 
in other environments. I'll see what I can do when I'm again at GPU 
machine tomorrow (parallel GPU debugging via SSH is not great...)

Best regards,
Karli




From triou at cea.fr  Tue Jan 14 10:15:06 2014
From: triou at cea.fr (Projet_TRIOU)
Date: Tue, 14 Jan 2014 17:15:06 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
 build)
In-Reply-To: <52D55AB8.9060301@mcs.anl.gov>
References: <52D54CD0.6070608@cea.fr> <52D55AB8.9060301@mcs.anl.gov>
Message-ID: <52D5628A.7080204@cea.fr>

On 01/14/14 16:41, Karl Rupp wrote:
> Hi Pierre,
>
> > I try running in parallel the ex19 test case on CPU and GPU:
>>
>> OPT="_opt"
>> dir=/ccc/cont002/home/den/triou/Version_test_airain-hybrid/Trio_U/lib/src/LIBPETSC/petsc/linux$OPT/src/snes/examples/tutorials 
>>
>>
>> option="-pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol
>> 1.e-5 -log_summary -ksp_view -cuda_show_devices"
>> mpirun -np 2 $dir/ex19 $option 1>cpu$OPT.log 2>&1
>> mpirun -np 2 $dir/ex19 -dm_vec_type cusp -dm_mat_type aijcusp $option
>> 1>gpu$OPT.log 2>&1
>>
>> With OPT="", PETSc optimized library is used, parallel calculation runs
>> well on CPU and GPU.
>> With OPT="_opt", PETSc non optimized library is used, parallel
>> calculation crashes on GPU (it is OK on CPU).
>>
>> I join the log files. The only difference seems that PETSc-dev is built
>> with -O3 intead of -g...
>> I could try to rebuild PETSc with -O2 but do you have any idea of the
>> problem ?
>
> I could reproduce the problem and also get some uninitialized variable 
> warnings in Valgrind. The debug version detects these errors, hence 
> you only see the errors in the debug build. For the optimized build, 
> chances are good that the computed values are either wrong or may 
> become wrong in other environments. I'll see what I can do when I'm 
> again at GPU machine tomorrow (parallel GPU debugging via SSH is not 
> great...)
Sorry, I mean:

Parallel calculation on CPU or GPU run well with PETSc non optimized library
Parallel calculation on GPU crashes with PETSc optimized library (on CPU 
it is OK)

I could add that the "mpirun -np 1 ex19" runs well for all builds on CPU 
and GPU.

Pierre
>
> Best regards,
> Karli
>
>


-- 
*Trio_U support team*
Marthe ROUX (Saclay)
Pierre LEDAC (Grenoble)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/3a2fc14e/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 14 10:19:59 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 14 Jan 2014 17:19:59 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
 build)
In-Reply-To: <52D5628A.7080204@cea.fr>
References: <52D54CD0.6070608@cea.fr> <52D55AB8.9060301@mcs.anl.gov>
	<52D5628A.7080204@cea.fr>
Message-ID: <52D563AF.9080207@mcs.anl.gov>

Hi Pierre,

 >> I could reproduce the problem and also get some uninitialized variable
>> warnings in Valgrind. The debug version detects these errors, hence
>> you only see the errors in the debug build. For the optimized build,
>> chances are good that the computed values are either wrong or may
>> become wrong in other environments. I'll see what I can do when I'm
>> again at GPU machine tomorrow (parallel GPU debugging via SSH is not
>> great...)
> Sorry, I mean:
>
> Parallel calculation on CPU or GPU run well with PETSc non optimized library
> Parallel calculation on GPU crashes with PETSc optimized library (on CPU
> it is OK)

The fact that it happens to run in one mode out of {debug, optimized} 
but not in the other is at most a lucky coincidence, but it still means 
that this is a bug we need to solve :-)


> I could add that the "mpirun -np 1 ex19" runs well for all builds on CPU
> and GPU.

I see valgrind warnings in the vector scatter routines, which is likely 
the reason why it doesn't work with multiple MPI ranks.

Best regards,
Karli



From stefano.zampini at gmail.com  Tue Jan 14 10:20:05 2014
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Tue, 14 Jan 2014 17:20:05 +0100
Subject: [petsc-dev] PCBDDC
Message-ID: <CAGPUishSO=D3oWinCP8cqvZoqOrAh0LXrc-JSH5NXD5o9WQRbQ@mail.gmail.com>

Hi,

I think PCBDDC is now stable enough to be available without the configure
switch --with-pcbddc. Could you please review and merge my opened pull
requests (134, 135 and 136) and remove the configure switch together with
the int64 restriction on the PCBDDC package?

I would like to solve any issues coming from the nightly builds before the
next release. If it is not safe to do this, I'm waiting for suggestions.

Best
-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/080e96ef/attachment.html>

From paulmullowney at gmail.com  Tue Jan 14 10:21:22 2014
From: paulmullowney at gmail.com (Paul Mullowney)
Date: Tue, 14 Jan 2014 09:21:22 -0700
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
	build)
In-Reply-To: <52D563AF.9080207@mcs.anl.gov>
References: <52D54CD0.6070608@cea.fr> <52D55AB8.9060301@mcs.anl.gov>
	<52D5628A.7080204@cea.fr> <52D563AF.9080207@mcs.anl.gov>
Message-ID: <CAMJ8fwrsJYOHsvs9cPwhTeH_OinaKHMSTMo0RrJco7Qi2qWknQ@mail.gmail.com>

Previously, I had noticed strange behaviour when running the GPU code with
the threadComm package. It might be worth trying to disable that code in
the build to see if the problem persists?
-Paul


On Tue, Jan 14, 2014 at 9:19 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Pierre,
>
>
> >> I could reproduce the problem and also get some uninitialized variable
>
>> warnings in Valgrind. The debug version detects these errors, hence
>>> you only see the errors in the debug build. For the optimized build,
>>> chances are good that the computed values are either wrong or may
>>> become wrong in other environments. I'll see what I can do when I'm
>>> again at GPU machine tomorrow (parallel GPU debugging via SSH is not
>>> great...)
>>>
>> Sorry, I mean:
>>
>> Parallel calculation on CPU or GPU run well with PETSc non optimized
>> library
>> Parallel calculation on GPU crashes with PETSc optimized library (on CPU
>> it is OK)
>>
>
> The fact that it happens to run in one mode out of {debug, optimized} but
> not in the other is at most a lucky coincidence, but it still means that
> this is a bug we need to solve :-)
>
>
>
>  I could add that the "mpirun -np 1 ex19" runs well for all builds on CPU
>> and GPU.
>>
>
> I see valgrind warnings in the vector scatter routines, which is likely
> the reason why it doesn't work with multiple MPI ranks.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/4a931e92/attachment.html>

From balay at mcs.anl.gov  Tue Jan 14 10:55:43 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 14 Jan 2014 10:55:43 -0600
Subject: [petsc-dev] configuring on Titan
In-Reply-To: <CADOhEh4aBF=wwQSDGxVVo76J=UFus9C-JSwv6RioJSH7UZZGSQ@mail.gmail.com>
References: <CADOhEh4aBF=wwQSDGxVVo76J=UFus9C-JSwv6RioJSH7UZZGSQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401141054270.23986@asterix>

I pushed a fix to master.

If you rerun ./reconfigure-arch-titan-opt.py [with the option
--known-mpi-int64_t] - and it should not give this error.

Satish

On Tue, 14 Jan 2014, Mark Adams wrote:

> I'm trying to configure on Titan.  It tells me to run the compute node job
> but when I run the resulting reconfigure script it tells me to
> run conftest-arch-titan-opt again (infinite loop).  So I do:
> 
> mv ./conftest-arch-titan-opt $MEMBERWORK/env003/
> 
> adams/env003> aprun -n 1 ./conftest-arch-titan-opt
> 
> mv $MEMBERWORK/env003/reconfigure-arch-titan-opt.py .
> ./reconfigure-arch-titan-opt.py
> vi ./reconfigure-arch-titan-opt.py
> ./reconfigure-arch-titan-opt.py
> 
> I need to edit reconfigure:
> 
> *******************************************************************************
>                 ERROR in COMMAND LINE ARGUMENT to ./configure
> -------------------------------------------------------------------------------
> The option --known-mpi-int64_t should probably be --known-mpi-int64-t
> 
> I just delete this line.  The correct one seems to be there also.
> 
> Mark
> 



From bsmith at mcs.anl.gov  Tue Jan 14 13:16:17 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 14 Jan 2014 13:16:17 -0600
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <alpine.LFD.2.11.1401141036580.23986@asterix>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
Message-ID: <C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>


    Hmm, if the user provides --with-cc='win32fe cl?  shouldn?t we automatically add --with-cxx='win32fe cl? as well?

It does not harm (right?) and will prevent this from hitting other people in the future.

   Barry

Request-assigned: balay automatically add --with-cxx='win32fe cl?  if --with-cc='win32fe cl? is provided?


On Jan 14, 2014, at 10:37 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> Try adding configure option: --with-cxx='win32fe cl'
> 
> Satish
> 
> On Tue, 14 Jan 2014, Abilash Nair wrote:
> 
>> Hello all,
>> 
>> I am trying to build petsc to use as a library on a visual C++ project.
>> 
>> I configured the petsc (v3.4.3) with VC++ 2012 under the shell provided by
>> VC2012 with the following command (in cygwin):
>> 
>> ./config/configure.py --with-mpi=0 --download-f2cblaslapack=1 --with-fc=0
>> --with-cc='win32fe cl' --with-debugging=1
>> 
>> then I did a make all; make test while still in cygwin. I have sent the log
>> file outputs from these commands. As the files show, I am able to make with
>> no errors at all, however I am unable to execute any of the tests (error
>> being that a libstcc++.lib not found).
>> 
>> I hope you would be able to suggest a path forward.
>> 
>> Thanks!
>> Abilash.
>> 
> 



From aron at ahmadia.net  Tue Jan 14 13:21:55 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Tue, 14 Jan 2014 14:21:55 -0500
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
	<C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
Message-ID: <CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>

I'd make it a warning, not a default.


On Tue, Jan 14, 2014 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>     Hmm, if the user provides --with-cc='win32fe cl?  shouldn?t we
> automatically add --with-cxx='win32fe cl? as well?
>
> It does not harm (right?) and will prevent this from hitting other people
> in the future.
>
>    Barry
>
> Request-assigned: balay automatically add --with-cxx='win32fe cl?  if
> --with-cc='win32fe cl? is provided?
>
>
> On Jan 14, 2014, at 10:37 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>
> > Try adding configure option: --with-cxx='win32fe cl'
> >
> > Satish
> >
> > On Tue, 14 Jan 2014, Abilash Nair wrote:
> >
> >> Hello all,
> >>
> >> I am trying to build petsc to use as a library on a visual C++ project.
> >>
> >> I configured the petsc (v3.4.3) with VC++ 2012 under the shell provided
> by
> >> VC2012 with the following command (in cygwin):
> >>
> >> ./config/configure.py --with-mpi=0 --download-f2cblaslapack=1
> --with-fc=0
> >> --with-cc='win32fe cl' --with-debugging=1
> >>
> >> then I did a make all; make test while still in cygwin. I have sent the
> log
> >> file outputs from these commands. As the files show, I am able to make
> with
> >> no errors at all, however I am unable to execute any of the tests (error
> >> being that a libstcc++.lib not found).
> >>
> >> I hope you would be able to suggest a path forward.
> >>
> >> Thanks!
> >> Abilash.
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/e80916f8/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan 14 13:43:08 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 14 Jan 2014 13:43:08 -0600
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
	<C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
	<CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>
Message-ID: <F16F31AE-4D28-4C8B-A600-77B98DE083BA@mcs.anl.gov>


On Jan 14, 2014, at 1:21 PM, Aron Ahmadia <aron at ahmadia.net> wrote:

> I'd make it a warning, not a default.

  Why?

  We generally don?t produce ?warnings? during configuration currently

   Barry

> 
> 
> On Tue, Jan 14, 2014 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>     Hmm, if the user provides --with-cc='win32fe cl?  shouldn?t we automatically add --with-cxx='win32fe cl? as well?
> 
> It does not harm (right?) and will prevent this from hitting other people in the future.
> 
>    Barry
> 
> Request-assigned: balay automatically add --with-cxx='win32fe cl?  if --with-cc='win32fe cl? is provided?
> 
> 
> On Jan 14, 2014, at 10:37 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > Try adding configure option: --with-cxx='win32fe cl'
> >
> > Satish
> >
> > On Tue, 14 Jan 2014, Abilash Nair wrote:
> >
> >> Hello all,
> >>
> >> I am trying to build petsc to use as a library on a visual C++ project.
> >>
> >> I configured the petsc (v3.4.3) with VC++ 2012 under the shell provided by
> >> VC2012 with the following command (in cygwin):
> >>
> >> ./config/configure.py --with-mpi=0 --download-f2cblaslapack=1 --with-fc=0
> >> --with-cc='win32fe cl' --with-debugging=1
> >>
> >> then I did a make all; make test while still in cygwin. I have sent the log
> >> file outputs from these commands. As the files show, I am able to make with
> >> no errors at all, however I am unable to execute any of the tests (error
> >> being that a libstcc++.lib not found).
> >>
> >> I hope you would be able to suggest a path forward.
> >>
> >> Thanks!
> >> Abilash.
> >>
> >
> 
> 



From aron at ahmadia.net  Tue Jan 14 13:46:43 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Tue, 14 Jan 2014 14:46:43 -0500
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <F16F31AE-4D28-4C8B-A600-77B98DE083BA@mcs.anl.gov>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
	<C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
	<CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>
	<F16F31AE-4D28-4C8B-A600-77B98DE083BA@mcs.anl.gov>
Message-ID: <CAPhiW4jJ2R7k4+27DyzPDaVMz_8oRuM_Di2usqGp8++19jj1Ew@mail.gmail.com>

You produce warnings all the time.  When you ignore environment variables,
or another variable is set that suggests a mistake in configuration.

Or am I missing something here.


On Tue, Jan 14, 2014 at 2:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Jan 14, 2014, at 1:21 PM, Aron Ahmadia <aron at ahmadia.net> wrote:
>
> > I'd make it a warning, not a default.
>
>   Why?
>
>   We generally don?t produce ?warnings? during configuration currently
>
>    Barry
>
> >
> >
> > On Tue, Jan 14, 2014 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Hmm, if the user provides --with-cc='win32fe cl?  shouldn?t we
> automatically add --with-cxx='win32fe cl? as well?
> >
> > It does not harm (right?) and will prevent this from hitting other
> people in the future.
> >
> >    Barry
> >
> > Request-assigned: balay automatically add --with-cxx='win32fe cl?  if
> --with-cc='win32fe cl? is provided?
> >
> >
> > On Jan 14, 2014, at 10:37 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > Try adding configure option: --with-cxx='win32fe cl'
> > >
> > > Satish
> > >
> > > On Tue, 14 Jan 2014, Abilash Nair wrote:
> > >
> > >> Hello all,
> > >>
> > >> I am trying to build petsc to use as a library on a visual C++
> project.
> > >>
> > >> I configured the petsc (v3.4.3) with VC++ 2012 under the shell
> provided by
> > >> VC2012 with the following command (in cygwin):
> > >>
> > >> ./config/configure.py --with-mpi=0 --download-f2cblaslapack=1
> --with-fc=0
> > >> --with-cc='win32fe cl' --with-debugging=1
> > >>
> > >> then I did a make all; make test while still in cygwin. I have sent
> the log
> > >> file outputs from these commands. As the files show, I am able to
> make with
> > >> no errors at all, however I am unable to execute any of the tests
> (error
> > >> being that a libstcc++.lib not found).
> > >>
> > >> I hope you would be able to suggest a path forward.
> > >>
> > >> Thanks!
> > >> Abilash.
> > >>
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/96e9fadf/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan 14 14:01:01 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 14 Jan 2014 14:01:01 -0600
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <CAPhiW4jJ2R7k4+27DyzPDaVMz_8oRuM_Di2usqGp8++19jj1Ew@mail.gmail.com>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
	<C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
	<CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>
	<F16F31AE-4D28-4C8B-A600-77B98DE083BA@mcs.anl.gov>
	<CAPhiW4jJ2R7k4+27DyzPDaVMz_8oRuM_Di2usqGp8++19jj1Ew@mail.gmail.com>
Message-ID: <AC7CED3F-01C1-4CC3-86BB-76E3F22A1B13@mcs.anl.gov>


On Jan 14, 2014, at 1:46 PM, Aron Ahmadia <aron at ahmadia.net> wrote:

> You produce warnings all the time.  When you ignore environment variables, or another variable is set that suggests a mistake in configuration.

   Those few warnings all come in the initial start up of ./configure.  Maybe it doesn?t matter but I fear if we spew many warnings they start to get ignored. 

   But why shouldn?t we default to setting the cxx compiler correctly? We currently try to do it for non-Windows systems.

   Barry

> 
> Or am I missing something here.
> 
> 
> On Tue, Jan 14, 2014 at 2:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Jan 14, 2014, at 1:21 PM, Aron Ahmadia <aron at ahmadia.net> wrote:
> 
> > I'd make it a warning, not a default.
> 
>   Why?
> 
>   We generally don?t produce ?warnings? during configuration currently
> 
>    Barry
> 
> >
> >
> > On Tue, Jan 14, 2014 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Hmm, if the user provides --with-cc='win32fe cl?  shouldn?t we automatically add --with-cxx='win32fe cl? as well?
> >
> > It does not harm (right?) and will prevent this from hitting other people in the future.
> >
> >    Barry
> >
> > Request-assigned: balay automatically add --with-cxx='win32fe cl?  if --with-cc='win32fe cl? is provided?
> >
> >
> > On Jan 14, 2014, at 10:37 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > Try adding configure option: --with-cxx='win32fe cl'
> > >
> > > Satish
> > >
> > > On Tue, 14 Jan 2014, Abilash Nair wrote:
> > >
> > >> Hello all,
> > >>
> > >> I am trying to build petsc to use as a library on a visual C++ project.
> > >>
> > >> I configured the petsc (v3.4.3) with VC++ 2012 under the shell provided by
> > >> VC2012 with the following command (in cygwin):
> > >>
> > >> ./config/configure.py --with-mpi=0 --download-f2cblaslapack=1 --with-fc=0
> > >> --with-cc='win32fe cl' --with-debugging=1
> > >>
> > >> then I did a make all; make test while still in cygwin. I have sent the log
> > >> file outputs from these commands. As the files show, I am able to make with
> > >> no errors at all, however I am unable to execute any of the tests (error
> > >> being that a libstcc++.lib not found).
> > >>
> > >> I hope you would be able to suggest a path forward.
> > >>
> > >> Thanks!
> > >> Abilash.
> > >>
> > >
> >
> >
> 
> 



From mc0710 at gmail.com  Tue Jan 14 14:19:49 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 14 Jan 2014 14:19:49 -0600
Subject: [petsc-dev] SNES ex19 not using GPU despite passing the options
In-Reply-To: <52D504A3.502@mcs.anl.gov>
References: <CAJzU6sBQFB+VWeqo4A_GW9vo0TOgF12FYSCURx7W3EPA+y5jRw@mail.gmail.com>
	<52D504A3.502@mcs.anl.gov>
Message-ID: <CAJzU6sAY75UPW=fmZm_+uWsiL9kc+ahKEGWoBzbiMHOQqkfAmg@mail.gmail.com>

Hi Karli,

Thanks for the reply. That fixed it. I get only a 10% speed up using the
cusp options. Is the residual evaluation at each iteration happening on the
CPU or the GPU? Is there anyway one can do the residual evaluation on the
GPU too, after the data has been transferred? Ex42 shows how it can be done
using cusp but it looks really ugly and I want to use OpenCL. Basically can
I do something like this?

DMGetLocalVector(da, &localX); //Vector is now in GPU.
DMDAVecGetArray(da, localX, &x); //Array is on GPU.

//Create buffers for OpenCL
buffer = cl::Buffer(context, CL_MEM_USE_HOST_PTR |
                                                CL_MEM_READ_WRITE,
                                  sizeofarray, &x[X2Start-Ng][X1Start-Ng]
                                   , &clErr);

(I'm hoping that here CL_MEM_USE_HOST_PTR will give a pointer to the data
already on the GPU)

// Launch OpenCL kernels and now map the buffers to read off the data.

DMDAVecRestoreArray(da, localX, &x);
DMRestoreLocalVector(da, &localX);

I think the question is whether DMDAVecGetArray will return a pointer to
the data on the GPU or not.

Cheers,
Mani


On Tue, Jan 14, 2014 at 3:34 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
> the options are misspelled, you want to use
> -dm_vec_type cusp -dm_mat_type aijcusp
> (you can omit the 'mpi' to support both the MPI case and the sequential
> case).
>
> I also suggest doing all development in debug mode, in which case you
> would have received the following hints:
>
> WARNING! There are options you set that were not used!
> WARNING! could be spelling mistake, etc!
> Option left: name:-da_mat_type value: mpiaijcusp
> Option left: name:-da_vec_type value: mpicusp
> Option left: name:-dmmg_nlevels value: 1
> Option left: name:-preload value: off
>
> Best regards,
> Karli
>
>
>
>
> On 01/14/2014 04:28 AM, Mani Chandra wrote:
>
>> Hi,
>>
>> I tried to run SNES ex19 with the following options but it does not seem
>> to use my GPU. See the log summary attached. Am I interpretting the log
>> summary wrong? I don't see any CUSP calls to copy data from the CPU to
>> the GPU.
>>
>> /ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none
>> -dmmg_nlevels 1 -da_grid_x 300 -da_grid_y 300 -log_summary -mat_no_inode
>> -preload off  -cusp_synchronize -cuda_s
>> et_device 0
>>
>> I get the following output when I do -cuda_show_devices
>> CUDA device 0: Quadro FX 1800M
>>
>> Cheers,
>> Mani
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/5d9e71e4/attachment.html>

From aron at ahmadia.net  Tue Jan 14 14:45:15 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Tue, 14 Jan 2014 15:45:15 -0500
Subject: [petsc-dev] [petsc-users] PETSc/VC12 compilation
In-Reply-To: <AC7CED3F-01C1-4CC3-86BB-76E3F22A1B13@mcs.anl.gov>
References: <CAJTdXfeSfgBGAMi8ABh=x2RLkiS8d31_qrR4Mb1YETmik6iRow@mail.gmail.com>
	<alpine.LFD.2.11.1401141036580.23986@asterix>
	<C94E0484-AB69-4E22-9C05-58C23E6B68AB@mcs.anl.gov>
	<CAPhiW4jhvXRfC-+V7gTO5W6PWekd3hH0TejaDyhRngYt874kbA@mail.gmail.com>
	<F16F31AE-4D28-4C8B-A600-77B98DE083BA@mcs.anl.gov>
	<CAPhiW4jJ2R7k4+27DyzPDaVMz_8oRuM_Di2usqGp8++19jj1Ew@mail.gmail.com>
	<AC7CED3F-01C1-4CC3-86BB-76E3F22A1B13@mcs.anl.gov>
Message-ID: <CAPhiW4hwu+53b2UMAPX6dY7M7nEmVQk6hAuutERGGsmYWs2eRQ@mail.gmail.com>

On Tue, Jan 14, 2014 at 3:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> But why shouldn?t we default to setting the cxx compiler correctly? We
> currently try to do it for non-Windows systems.


I didn't realize this.  By all means do whatever you do elsewhere :)  I'd
appreciate a warning that you're doing it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140114/6d5b1fc7/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 14 16:49:59 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 14 Jan 2014 23:49:59 +0100
Subject: [petsc-dev] SNES ex19 not using GPU despite passing the options
In-Reply-To: <CAJzU6sAY75UPW=fmZm_+uWsiL9kc+ahKEGWoBzbiMHOQqkfAmg@mail.gmail.com>
References: <CAJzU6sBQFB+VWeqo4A_GW9vo0TOgF12FYSCURx7W3EPA+y5jRw@mail.gmail.com>	<52D504A3.502@mcs.anl.gov>
	<CAJzU6sAY75UPW=fmZm_+uWsiL9kc+ahKEGWoBzbiMHOQqkfAmg@mail.gmail.com>
Message-ID: <52D5BF17.3090403@mcs.anl.gov>

Hi Mani,

 > Thanks for the reply. That fixed it. I get only a 10% speed up using the
> cusp options. Is the residual evaluation at each iteration happening on
> the CPU or the GPU?

The residual evaluation happens on the CPU unless there is a dedicated 
kernel provided for this (which is not the case in ex19)


> Is there anyway one can do the residual evaluation
> on the GPU too, after the data has been transferred?

Technically it is possible by extracting the underlying GPU buffers from 
the vector objects and by manually managing the Field data. Frankly I 
don't know about the current state of the local-to-global mappings, you 
likely have to do quite some copying of data between host and device 
manually.


> Ex42 shows how it
> can be done using cusp but it looks really ugly and I want to use
> OpenCL. Basically can I do something like this?
>
> DMGetLocalVector(da, &localX); //Vector is now in GPU.
> DMDAVecGetArray(da, localX, &x); //Array is on GPU.
>
> //Create buffers for OpenCL
> buffer = cl::Buffer(context, CL_MEM_USE_HOST_PTR |
>                                                  CL_MEM_READ_WRITE,
>                                    sizeofarray, &x[X2Start-Ng][X1Start-Ng]
>                                     , &clErr);
>
> (I'm hoping that here CL_MEM_USE_HOST_PTR will give a pointer to the
> data already on the GPU)
>
> // Launch OpenCL kernels and now map the buffers to read off the data.
>
> DMDAVecRestoreArray(da, localX, &x);
> DMRestoreLocalVector(da, &localX);
>
> I think the question is whether DMDAVecGetArray will return a pointer to
> the data on the GPU or not.

*VecGetArray() will always return a pointer due to the inability to 
overload functions in C. Buffers in OpenCL are of type cl_mem, so this 
won't work. Also, you won't be able to copy a two-dimensional array with 
just one pointer &x[][]. As far as I know, we don't have any API which 
provides GPU buffers directly, but maybe Matt added some functions for 
this to work with FEM recently.

As far as I can tell, only providing the kernel won't suffice because we 
don't have the GPU-implementations for 'Field' data available. Hence, 
you would have to copy the x and b arrays manually and then copy 
everything back, which is most likely too much of a performance hit to 
be worth the effort. Since GPUs are getting more and more integrated 
into CPUs, it's questionable whether it's worth the time to implement 
such additional memory management for accelerators if they disappear in 
their discrete PCI-Express form in a few years from now...

Best regards,
Karli



From michael.lange at imperial.ac.uk  Wed Jan 15 07:16:00 2014
From: michael.lange at imperial.ac.uk (Michael Lange)
Date: Wed, 15 Jan 2014 13:16:00 +0000
Subject: [petsc-dev] Comp-comm overlap with DMPlex
Message-ID: <52D68A10.6040605@imperial.ac.uk>

Hi,

I am trying to implement a computation-communication overlap with 
DMPlex, where local values that do not need to be sent are updated while 
the DM/SF is broadcasting the ghost values (DMGlobalToLocal() or 
PetscSFBcast(), I've tried both). The problem is that this only works if 
I force the broadcast to end before the computation is performed, but it 
fails if they actually overlap. In that case it seems that the updates 
to the global Vec are ignored completely. Am I missing something here or 
can somebody point me to a working example that does comp-comm overlap 
with DMPlex?

Kind regards
Michael Lange


From knepley at gmail.com  Wed Jan 15 08:41:24 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 15 Jan 2014 08:41:24 -0600
Subject: [petsc-dev] Comp-comm overlap with DMPlex
In-Reply-To: <52D68A10.6040605@imperial.ac.uk>
References: <52D68A10.6040605@imperial.ac.uk>
Message-ID: <CAMYG4G=b7b24KHvCJbWs4R3Rzo378F0Wq+evr=CcnxY2+vETnQ@mail.gmail.com>

On Wed, Jan 15, 2014 at 7:16 AM, Michael Lange <michael.lange at imperial.ac.uk
> wrote:

> Hi,
>
> I am trying to implement a computation-communication overlap with DMPlex,
> where local values that do not need to be sent are updated while the DM/SF
> is broadcasting the ghost values (DMGlobalToLocal() or PetscSFBcast(), I've
> tried both). The problem is that this only works if I force the broadcast
> to end before the computation is performed, but it fails if they actually
> overlap. In that case it seems that the updates to the global Vec are
> ignored completely. Am I missing something here or can somebody point me to
> a working example that does comp-comm overlap with DMPlex?
>

0) You would first need to divide the computation units (cells for FEM)
into boundary and interior, which I do not currently do. Then you can

     DMLocalToLocalBegin()
    <compute interior>
    DMLocalToLocalEnd()
    <compute boundary>

1) DMGlobalToLocal() copies the entire local vector, so it is not really
what you want. You want LocalToLocal(), which I do not currently
    make. Its not hard. You take the GlobalToLocal map and remove
everything that is not on the boundary, meaning everything in the SF
    that comes from the same proc. So you make another SF, and use the same
PetscSFBcast() in DMLocalToLocal().

    Matt


> Kind regards
> Michael Lange
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/b61c9950/attachment.html>

From triou at cea.fr  Wed Jan 15 09:17:07 2014
From: triou at cea.fr (Projet_TRIOU)
Date: Wed, 15 Jan 2014 16:17:07 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
 build)
In-Reply-To: <CAMJ8fwrsJYOHsvs9cPwhTeH_OinaKHMSTMo0RrJco7Qi2qWknQ@mail.gmail.com>
References: <52D54CD0.6070608@cea.fr>	<52D55AB8.9060301@mcs.anl.gov>	<52D5628A.7080204@cea.fr>	<52D563AF.9080207@mcs.anl.gov>
	<CAMJ8fwrsJYOHsvs9cPwhTeH_OinaKHMSTMo0RrJco7Qi2qWknQ@mail.gmail.com>
Message-ID: <52D6A673.7040202@cea.fr>

I tried to rebuild the optimized PETSc library by changing several 
options and ran:

mpirun -np 2 ./ex19 -cuda_show_devices -dm_mat_type aijcusp -dm_vec_type 
cusp
-ksp_type fgmres -ksp_view -log_summary -pc_type none
-snes_monitor_short -snes_rtol 1.e-5

Options used:
--with-pthread=1 -O3  -> crash
--with-pthread=0 -O2  -> crash
--with-debugging=1 --with-pthread=1 -O2 -> OK

So --with-debugging=1 is the key to avoid the crash. Not
good for the performance of course...

If it can helps,

Pierre

> Previously, I had noticed strange behaviour when running the GPU code 
> with the threadComm package. It might be worth trying to disable that 
> code in the build to see if the problem persists?
> -Paul
>
>
> On Tue, Jan 14, 2014 at 9:19 AM, Karl Rupp <rupp at mcs.anl.gov 
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Pierre,
>
>
>     >> I could reproduce the problem and also get some uninitialized
>     variable
>
>             warnings in Valgrind. The debug version detects these
>             errors, hence
>             you only see the errors in the debug build. For the
>             optimized build,
>             chances are good that the computed values are either wrong
>             or may
>             become wrong in other environments. I'll see what I can do
>             when I'm
>             again at GPU machine tomorrow (parallel GPU debugging via
>             SSH is not
>             great...)
>
>         Sorry, I mean:
>
>         Parallel calculation on CPU or GPU run well with PETSc non
>         optimized library
>         Parallel calculation on GPU crashes with PETSc optimized
>         library (on CPU
>         it is OK)
>
>
>     The fact that it happens to run in one mode out of {debug,
>     optimized} but not in the other is at most a lucky coincidence,
>     but it still means that this is a bug we need to solve :-)
>
>
>
>         I could add that the "mpirun -np 1 ex19" runs well for all
>         builds on CPU
>         and GPU.
>
>
>     I see valgrind warnings in the vector scatter routines, which is
>     likely the reason why it doesn't work with multiple MPI ranks.
>
>     Best regards,
>     Karli
>
>


-- 
*Trio_U support team*
Marthe ROUX (Saclay)
Pierre LEDAC (Grenoble)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_2_gpu_linux_opt.log
Type: text/x-log
Size: 14966 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_2_gpu_linux.log
Type: text/x-log
Size: 14946 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_2_cpu_linux_opt.log
Type: text/x-log
Size: 15780 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_2_cpu_linux.log
Type: text/x-log
Size: 15761 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_1_gpu_linux_opt.log
Type: text/x-log
Size: 16021 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_1_gpu_linux.log
Type: text/x-log
Size: 16012 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_1_cpu_linux_opt.log
Type: text/x-log
Size: 15694 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_np_1_cpu_linux.log
Type: text/x-log
Size: 15675 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/df605b9a/attachment-0007.bin>

From rupp at mcs.anl.gov  Wed Jan 15 09:41:39 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 15 Jan 2014 16:41:39 +0100
Subject: [petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized
 build)
In-Reply-To: <52D6A673.7040202@cea.fr>
References: <52D54CD0.6070608@cea.fr>	<52D55AB8.9060301@mcs.anl.gov>	<52D5628A.7080204@cea.fr>	<52D563AF.9080207@mcs.anl.gov>
	<CAMJ8fwrsJYOHsvs9cPwhTeH_OinaKHMSTMo0RrJco7Qi2qWknQ@mail.gmail.com>
	<52D6A673.7040202@cea.fr>
Message-ID: <52D6AC33.30804@mcs.anl.gov>

Hi Pierre,

 > I tried to rebuild the optimized PETSc library by changing several
> options and ran:
>
> mpirun -np 2 ./ex19 -cuda_show_devices -dm_mat_type aijcusp -dm_vec_type
> cusp
> -ksp_type fgmres -ksp_view -log_summary -pc_type none
> -snes_monitor_short -snes_rtol 1.e-5
>
> Options used:
> --with-pthread=1 -O3  -> crash
> --with-pthread=0 -O2  -> crash
> --with-debugging=1 --with-pthread=1 -O2 -> OK
>
> So --with-debugging=1 is the key to avoid the crash. Not
> good for the performance of course...

thanks for the input. I get crashes even when using --with-debugging=1, 
and valgrind spits out a couple of errors as soon as
   -dm_vec_type cusp
is provided. I'll keep digging, the error is somewhere in the VecScatter 
routines when using CUSP...

Best regards,
Karli


>
> If it can helps,
>
> Pierre
>
>> Previously, I had noticed strange behaviour when running the GPU code
>> with the threadComm package. It might be worth trying to disable that
>> code in the build to see if the problem persists?
>> -Paul
>>
>>
>> On Tue, Jan 14, 2014 at 9:19 AM, Karl Rupp <rupp at mcs.anl.gov
>> <mailto:rupp at mcs.anl.gov>> wrote:
>>
>>     Hi Pierre,
>>
>>
>>     >> I could reproduce the problem and also get some uninitialized
>>     variable
>>
>>             warnings in Valgrind. The debug version detects these
>>             errors, hence
>>             you only see the errors in the debug build. For the
>>             optimized build,
>>             chances are good that the computed values are either wrong
>>             or may
>>             become wrong in other environments. I'll see what I can do
>>             when I'm
>>             again at GPU machine tomorrow (parallel GPU debugging via
>>             SSH is not
>>             great...)
>>
>>         Sorry, I mean:
>>
>>         Parallel calculation on CPU or GPU run well with PETSc non
>>         optimized library
>>         Parallel calculation on GPU crashes with PETSc optimized
>>         library (on CPU
>>         it is OK)
>>
>>
>>     The fact that it happens to run in one mode out of {debug,
>>     optimized} but not in the other is at most a lucky coincidence,
>>     but it still means that this is a bug we need to solve :-)
>>
>>
>>
>>         I could add that the "mpirun -np 1 ex19" runs well for all
>>         builds on CPU
>>         and GPU.
>>
>>
>>     I see valgrind warnings in the vector scatter routines, which is
>>     likely the reason why it doesn't work with multiple MPI ranks.
>>
>>     Best regards,
>>     Karli
>>
>>
>
>
> --
> *Trio_U support team*
> Marthe ROUX (Saclay)
> Pierre LEDAC (Grenoble)



From knepley at gmail.com  Wed Jan 15 11:20:37 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 15 Jan 2014 11:20:37 -0600
Subject: [petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a
 parallel calculation with PETSc>3.3
In-Reply-To: <52D677C5.90204@cea.fr>
References: <52D677C5.90204@cea.fr>
Message-ID: <CAMYG4G=G6Oa=ph3qyQoYCjTh1-0fG4UF9S3cAHdBpRh1MB5+tQ@mail.gmail.com>

This is a bug in the setup phase for Mat. I did not put in the logic, so I
might be missing the point. Here is
my outline:

  1) When MatSetPreallocation_*() is called, we set the
NEW_NONZERO_ALLOCATION_ERR option, but ONLY
       if the user passed a nonzero nz or nnz.

  2) During assembly, we check A->B->nonew, and do a collective call if it
is 0, meaning NEW_NONZERO_LOCATIONS

I understand that we would like to avoid collective calls in MatAssembly(),
but to guarantee correctness then we would
need to make the option collective, which is not true right now. I see

  Slightly Dangerous: Change SetPreallocation to make the option value
collective

  Slightly Slow: Always check for B disassembly

What do you think?

   Matt

---------- Forwarded message ----------
From: Projet_TRIOU <triou at cea.fr>
Date: Wed, Jan 15, 2014 at 5:57 AM
Subject: [petsc-users] MatAssemblyEnd hangs during a parallel calculation
with PETSc>3.3
To: petsc-users at mcs.anl.gov


Hi all,

I switched from PETSc 3.3 to PETSc 3.4.3 and all was fine except for
one of my test case on 3 processors where one processor was
dealing an empty local part of the global matrix.

My code hangs just during the call at MatAssemblyEnd:

ierr = MatMPIAIJSetPreallocation(MatricePetsc, PETSC_DEFAULT, d_nnz.addr(),
PETSC_DEFAULT, o_nnz.addr());
...
ierr = MatAssemblyEnd(MatricePetsc, MAT_FINAL_ASSEMBLY);

When I debugged, I notice on the empty processor, that in
src/mat/impls/aij/mpi/mpiaij.c:

  if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
    ierr = MPI_Allreduce(&mat->was_assembled,&other_disassembled,
1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
    if (mat->was_assembled && !other_disassembled) {
      ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
    }

((Mat_SeqAIJ*)aij->B->data)->nonew was 0 on the "empty" processor
and -2 on the 2 others...

I bypassed my problem with a different call to MatMPIAIJSetPreallocation():

      if (nb_rows==0) // Local matrix is empty
         ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0,
NULL, 0, NULL);
      else
         ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_,
PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());

Now, it runs well. So I don't know if it is a PETSc regression or if I was
abusively
calling MatMPISBAIJSetPreallocation with d_nnz/o_nnz empty arrays....

Thanks,

Pierre




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/35d3d828/attachment.html>

From aron at ahmadia.net  Wed Jan 15 12:14:21 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Wed, 15 Jan 2014 13:14:21 -0500
Subject: [petsc-dev] Include what you use
Message-ID: <CAPhiW4gh8W8RukZyAZhWySARzpm5yST3E5+UgzrD60GUyPKMvQ@mail.gmail.com>

This on your radar?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/e9a94790/attachment.html>

From aron at ahmadia.net  Wed Jan 15 12:14:35 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Wed, 15 Jan 2014 13:14:35 -0500
Subject: [petsc-dev] Include what you use
In-Reply-To: <CAPhiW4gh8W8RukZyAZhWySARzpm5yST3E5+UgzrD60GUyPKMvQ@mail.gmail.com>
References: <CAPhiW4gh8W8RukZyAZhWySARzpm5yST3E5+UgzrD60GUyPKMvQ@mail.gmail.com>
Message-ID: <CAPhiW4iywP9_b4UgP7WZneHn0MMp0F+BPvYoTnHc2sHVMBTFvQ@mail.gmail.com>

Stupid GMail.  Link:

http://sylvestre.ledru.info/blog/2013/09/08/include-what-you-use-yet

A


On Wed, Jan 15, 2014 at 1:14 PM, Aron Ahmadia <aron at ahmadia.net> wrote:

> This on your radar?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/2c77c65e/attachment.html>

From rupp at mcs.anl.gov  Wed Jan 15 12:52:49 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 15 Jan 2014 19:52:49 +0100
Subject: [petsc-dev] Include what you use
In-Reply-To: <CAPhiW4iywP9_b4UgP7WZneHn0MMp0F+BPvYoTnHc2sHVMBTFvQ@mail.gmail.com>
References: <CAPhiW4gh8W8RukZyAZhWySARzpm5yST3E5+UgzrD60GUyPKMvQ@mail.gmail.com>
	<CAPhiW4iywP9_b4UgP7WZneHn0MMp0F+BPvYoTnHc2sHVMBTFvQ@mail.gmail.com>
Message-ID: <52D6D901.8070506@mcs.anl.gov>

Hi Aron,

 > Stupid GMail.  Link:
>
> http://sylvestre.ledru.info/blog/2013/09/08/include-what-you-use-yet

Cool, thanks for the link!

Best regards,
Karli



From bsmith at mcs.anl.gov  Wed Jan 15 16:26:04 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 15 Jan 2014 16:26:04 -0600
Subject: [petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a
	parallel calculation with PETSc>3.3
In-Reply-To: <CAMYG4G=G6Oa=ph3qyQoYCjTh1-0fG4UF9S3cAHdBpRh1MB5+tQ@mail.gmail.com>
References: <52D677C5.90204@cea.fr>
	<CAMYG4G=G6Oa=ph3qyQoYCjTh1-0fG4UF9S3cAHdBpRh1MB5+tQ@mail.gmail.com>
Message-ID: <E182486B-EC9E-48BE-9F35-293B66463BB4@mcs.anl.gov>


On Jan 15, 2014, at 11:20 AM, Matthew Knepley <knepley at gmail.com> wrote:

> This is a bug in the setup phase for Mat. I did not put in the logic, so I might be missing the point. Here is
> my outline:
> 
>   1) When MatSetPreallocation_*() is called, we set the NEW_NONZERO_ALLOCATION_ERR option, but ONLY
>        if the user passed a nonzero nz or nnz.

   Matt,

   Ahhh. 

   The background is we wanted PETSc to automatically YELL if someone preallocated but did not preallocate enough. 

   But if the user did not preallocate we automatically call MatMPIXXSetPreallocation_MPIXXX() with default values that do not setup the yelling.

   If might be that some process has some rows, but they are all empty so our current model is totally incorrect

1)   We could do a global reduction in MatMPIXXXXSetPreallocation_MPIXXX() and if ANY process allocates space then mark the nonew.  We definitely don?t want to do a global reduction on every mat assembly.   For efficiency we could not automatically set the nonew for optimized builds but that is a dangerous game.

2)  Alternative, when MatMPIXXXXSetPreallocation_MPIXXX() is called, ALWAYS set the nonew flag, but when it is called because the user never called it like in

PetscErrorCode MatSetUp_MPIAIJ(Mat A)
{
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr =  MatMPIAIJSetPreallocation(A,PETSC_DEFAULT,0,PETSC_DEFAULT,0);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

we immediately change the flag after the call by added a MatSetOption() after the call to MatMPIXXXSetPreallocation()

I think 2 is ok and suggest doing that

   Barry


   

> 
>   2) During assembly, we check A->B->nonew, and do a collective call if it is 0, meaning NEW_NONZERO_LOCATIONS
> 
> I understand that we would like to avoid collective calls in MatAssembly(), but to guarantee correctness then we would
> need to make the option collective, which is not true right now. I see
> 
>   Slightly Dangerous: Change SetPreallocation to make the option value collective
> 
>   Slightly Slow: Always check for B disassembly
> 
> What do you think?
> 
>    Matt
> 
> ---------- Forwarded message ----------
> From: Projet_TRIOU <triou at cea.fr>
> Date: Wed, Jan 15, 2014 at 5:57 AM
> Subject: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3
> To: petsc-users at mcs.anl.gov
> 
> 
> Hi all,
> 
> I switched from PETSc 3.3 to PETSc 3.4.3 and all was fine except for
> one of my test case on 3 processors where one processor was
> dealing an empty local part of the global matrix.
> 
> My code hangs just during the call at MatAssemblyEnd:
> 
> ierr = MatMPIAIJSetPreallocation(MatricePetsc, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> ...
> ierr = MatAssemblyEnd(MatricePetsc, MAT_FINAL_ASSEMBLY);
> 
> When I debugged, I notice on the empty processor, that in
> src/mat/impls/aij/mpi/mpiaij.c:
> 
>   if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
>     ierr = MPI_Allreduce(&mat->was_assembled,&other_disassembled,1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
>     if (mat->was_assembled && !other_disassembled) {
>       ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
>     }
> 
> ((Mat_SeqAIJ*)aij->B->data)->nonew was 0 on the "empty" processor
> and -2 on the 2 others...
> 
> I bypassed my problem with a different call to MatMPIAIJSetPreallocation():
> 
>       if (nb_rows==0) // Local matrix is empty
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, NULL, 0, NULL);
>       else
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> 
> Now, it runs well. So I don't know if it is a PETSc regression or if I was abusively
> calling MatMPISBAIJSetPreallocation with d_nnz/o_nnz empty arrays....
> 
> Thanks,
> 
> Pierre
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From bsmith at mcs.anl.gov  Wed Jan 15 16:36:29 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 15 Jan 2014 16:36:29 -0600
Subject: [petsc-dev] Fwd: [petsc-users] Local size 7 not compatible with
	block size 13
References: <52D6DDFA.2010801@ornl.gov>
Message-ID: <C92F5F44-06FE-46EF-95AB-6F242F59C929@mcs.anl.gov>


   This problem comes up for MPIAIJ matrices when the ?off-diagona? matrix is compressed to only the  columns that have non zeros in them. When a MatDuplicate() is done on this it detects the inconsistency that the number of local columns may no longer be divisible by the block size. We could handle this by

1) when the matrix is compressed (removing empty columns) we just mark the block size to 1 if the new number of local columns is not divisible by the block size 

but then if the more values are put into the matrix when the matrix is uncompressed it no longer has the block size > 1 (possibly hurting efficiency) so we could

2) stash the old block size in the matrix and put it back when we uncompress it.

What do you think? Should I do 1)? 2)?

   Barry 

Request-assigned: Barry   fix MatDuplicate() problem on MPIAIJ matrices where block size of off diagonal matrix is no longer valid when compressed

Begin forwarded message:

> From: "Jay J. Billings" <billingsjj at ornl.gov>
> Subject: [petsc-users] Local size 7 not compatible with block size 13
> Date: January 15, 2014 at 1:14:02 PM CST
> To: <petsc-users at mcs.anl.gov>
> 
> Everyone,
> 
> I'm trying to run PETSc in parallel and I'm getting an error I haven't seen before that the local and block size arguments don't match. Can anyone point me in the right direction on this? It happens right after I call TSSolve and, according to the backtrace, when TSSolve is trying to form the Jacobian. It doesn't look like it is actually calling my Jacobian routine, although I could be wrong about that.
> 
> Jay
> 
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Arguments are incompatible!
> [1]PETSC ERROR: Arguments are incompatible!
> [0]PETSC ERROR: Local size 7 not compatible with block size 13!
> [1]PETSC ERROR: Local size 7 not compatible with block size 13!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> [1]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> [1]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> [1]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> [1]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [1]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [1]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [1]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> [1]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> [1]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> [0]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> [0]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> [0]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> [0]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> [0]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> [0]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> 
> -- 
> Jay Jay Billings
> Oak Ridge National Laboratory
> Twitter Handle: @jayjaybillings
> 



From knepley at gmail.com  Wed Jan 15 16:44:39 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 15 Jan 2014 16:44:39 -0600
Subject: [petsc-dev] Fwd: [petsc-users] Local size 7 not compatible with
 block size 13
In-Reply-To: <C92F5F44-06FE-46EF-95AB-6F242F59C929@mcs.anl.gov>
References: <52D6DDFA.2010801@ornl.gov>
	<C92F5F44-06FE-46EF-95AB-6F242F59C929@mcs.anl.gov>
Message-ID: <CAMYG4GmU4yXc0Jgeswy4YHwkC4yNxdPZyxmTxxnw5SOOJD+U6A@mail.gmail.com>

On Wed, Jan 15, 2014 at 4:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    This problem comes up for MPIAIJ matrices when the ?off-diagona? matrix
> is compressed to only the  columns that have non zeros in them. When a
> MatDuplicate() is done on this it detects the inconsistency that the number
> of local columns may no longer be divisible by the block size. We could
> handle this by
>
> 1) when the matrix is compressed (removing empty columns) we just mark the
> block size to 1 if the new number of local columns is not divisible by the
> block size
>
> but then if the more values are put into the matrix when the matrix is
> uncompressed it no longer has the block size > 1 (possibly hurting
> efficiency) so we could
>
> 2) stash the old block size in the matrix and put it back when we
> uncompress it.
>
> What do you think? Should I do 1)? 2)?
>

2) seems better, but I am worried about the complexity. I do not quite
understand the data structures either. I thought
B was just a MatSeqAIJ + column map. However, there is some
compression/decompression code I guess. What is
the compressed matrix, and where is the compressed flag?

   Matt


>    Barry
>
> Request-assigned: Barry   fix MatDuplicate() problem on MPIAIJ matrices
> where block size of off diagonal matrix is no longer valid when compressed
>
> Begin forwarded message:
>
> > From: "Jay J. Billings" <billingsjj at ornl.gov>
> > Subject: [petsc-users] Local size 7 not compatible with block size 13
> > Date: January 15, 2014 at 1:14:02 PM CST
> > To: <petsc-users at mcs.anl.gov>
> >
> > Everyone,
> >
> > I'm trying to run PETSc in parallel and I'm getting an error I haven't
> seen before that the local and block size arguments don't match. Can anyone
> point me in the right direction on this? It happens right after I call
> TSSolve and, according to the backtrace, when TSSolve is trying to form the
> Jacobian. It doesn't look like it is actually calling my Jacobian routine,
> although I could be wrong about that.
> >
> > Jay
> >
> > [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Arguments are incompatible!
> > [1]PETSC ERROR: Arguments are incompatible!
> > [0]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [1]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893
>  GIT Date: 2014-01-13 09:37:41 -0500
> > [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [1]PETSC ERROR: See docs/index.html for manual pages.
> > [1]PETSC ERROR:
> ------------------------------------------------------------------------
> > [1]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by
> jay Wed Jan 15 14:12:23 2014
> > [1]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [1]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [1]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1
> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no
> --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1
> --download-hypre=yes --with-debugging=0
> > [1]PETSC ERROR:
> ------------------------------------------------------------------------
> > [1]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in
> /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [1]PETSC ERROR: MatSetBlockSizes() line 6772 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in
> /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in
> /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: TSGetRHSMats_Private() line 605 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSComputeIJacobian() line 782 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in
> /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: SNESTSFormJacobian() line 3541 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESComputeJacobian() line 2253 in
> /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in
> /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [1]PETSC ERROR: SNESSolve() line 3812 in
> /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: TSStep_ARKIMEX() line 776 in
> /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: TSStep() line 2625 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSSolve() line 2741 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: checkPetscError() line 65 in
> /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl
> /xolotlSolver/PetscSolver.cpp
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893
>  GIT Date: 2014-01-13 09:37:41 -0500
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by
> jay Wed Jan 15 14:12:23 2014
> > [0]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [0]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [0]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1
> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no
> --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1
> --download-hypre=yes --with-debugging=0
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in
> /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [0]PETSC ERROR: MatSetBlockSizes() line 6772 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in
> /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in
> /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in
> /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: TSGetRHSMats_Private() line 605 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSComputeIJacobian() line 782 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in
> /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: SNESTSFormJacobian() line 3541 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESComputeJacobian() line 2253 in
> /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in
> /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [0]PETSC ERROR: SNESSolve() line 3812 in
> /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: TSStep_ARKIMEX() line 776 in
> /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: TSStep() line 2625 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSSolve() line 2741 in
> /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: checkPetscError() line 65 in
> /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl
> /xolotlSolver/PetscSolver.cpp
> >
> > --
> > Jay Jay Billings
> > Oak Ridge National Laboratory
> > Twitter Handle: @jayjaybillings
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140115/35c87b55/attachment.html>

From bsmith at mcs.anl.gov  Wed Jan 15 17:31:30 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 15 Jan 2014 17:31:30 -0600
Subject: [petsc-dev] Fwd: [petsc-users] Local size 7 not compatible with
	block size 13
In-Reply-To: <CAMYG4GmU4yXc0Jgeswy4YHwkC4yNxdPZyxmTxxnw5SOOJD+U6A@mail.gmail.com>
References: <52D6DDFA.2010801@ornl.gov>
	<C92F5F44-06FE-46EF-95AB-6F242F59C929@mcs.anl.gov>
	<CAMYG4GmU4yXc0Jgeswy4YHwkC4yNxdPZyxmTxxnw5SOOJD+U6A@mail.gmail.com>
Message-ID: <AA54F3BE-71A9-4793-B858-8BAFC5C71619@mcs.anl.gov>


On Jan 15, 2014, at 4:44 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 15, 2014 at 4:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    This problem comes up for MPIAIJ matrices when the ?off-diagona? matrix is compressed to only the  columns that have non zeros in them. When a MatDuplicate() is done on this it detects the inconsistency that the number of local columns may no longer be divisible by the block size. We could handle this by
> 
> 1) when the matrix is compressed (removing empty columns) we just mark the block size to 1 if the new number of local columns is not divisible by the block size
> 
> but then if the more values are put into the matrix when the matrix is uncompressed it no longer has the block size > 1 (possibly hurting efficiency) so we could
> 
> 2) stash the old block size in the matrix and put it back when we uncompress it.
> 
> What do you think? Should I do 1)? 2)?
> 
> 2) seems better, but I am worried about the complexity. I do not quite understand the data structures either. I thought
> B was just a MatSeqAIJ + column map. However, there is some compression/decompression code I guess. What is
> the compressed matrix,

    It is the same matrix, just the column indices are changed to shrink down on compress and go back up on non-compression.

> and where is the compressed flag?

   It doesn?t actually have a flag, the off-diagonal is always compressed on assembly. 

   I just looked again at the code and realized that 2) is actually already handled correctly. So I can fix 1) easily and will get to it soon.

  BTW: longer term issue is TS is keeping too many matrices around (memory hog).

    Barry

> 
>    Matt
>  
>    Barry
> 
> Request-assigned: Barry   fix MatDuplicate() problem on MPIAIJ matrices where block size of off diagonal matrix is no longer valid when compressed
> 
> Begin forwarded message:
> 
> > From: "Jay J. Billings" <billingsjj at ornl.gov>
> > Subject: [petsc-users] Local size 7 not compatible with block size 13
> > Date: January 15, 2014 at 1:14:02 PM CST
> > To: <petsc-users at mcs.anl.gov>
> >
> > Everyone,
> >
> > I'm trying to run PETSc in parallel and I'm getting an error I haven't seen before that the local and block size arguments don't match. Can anyone point me in the right direction on this? It happens right after I call TSSolve and, according to the backtrace, when TSSolve is trying to form the Jacobian. It doesn't look like it is actually calling my Jacobian routine, although I could be wrong about that.
> >
> > Jay
> >
> > [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Arguments are incompatible!
> > [1]PETSC ERROR: Arguments are incompatible!
> > [0]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [1]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> > [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [1]PETSC ERROR: See docs/index.html for manual pages.
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> > [1]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [1]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [1]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [1]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [1]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> > [0]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [0]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [0]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [0]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [0]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> >
> > --
> > Jay Jay Billings
> > Oak Ridge National Laboratory
> > Twitter Handle: @jayjaybillings
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From bsmith at mcs.anl.gov  Thu Jan 16 10:29:41 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 16 Jan 2014 10:29:41 -0600
Subject: [petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a
	parallel calculation with PETSc>3.3
References: <8738ko582c.fsf@jedbrown.org>
Message-ID: <4A1F82DF-A687-4876-8028-281BDFBDDE7E@mcs.anl.gov>


  Jed,

    I think it is better to make this safe. If user calls MatXXXSetPreallocation() then set nonew flag, if they do not call it then do not set nonew flag. Having it depend on the exact arguments passed in is just too dangerous.

    See my other mail to petsc-dev

when MatMPIXXXXSetPreallocation_MPIXXX() is called, ALWAYS set the nonew flag, but when it is called because the user never called it like in

PetscErrorCode MatSetUp_MPIAIJ(Mat A)
{
 PetscErrorCode ierr;

 PetscFunctionBegin;
 ierr =  MatMPIAIJSetPreallocation(A,PETSC_DEFAULT,0,PETSC_DEFAULT,0);CHKERRQ(ierr);
 PetscFunctionReturn(0);
}

we immediately change the flag after the call by added a MatSetOption() after the call to MatMPIXXXSetPreallocation()

Request-assigned:  Matt volunteered to fix. MatAssemblyEnd hangs during a parallel calculation do to preallocation specifics

Begin forwarded message:

> From: Jed Brown <jed at jedbrown.org>
> Subject: Re: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3
> Date: January 16, 2014 at 8:12:59 AM CST
> To: Projet_TRIOU <triou at cea.fr>, <petsc-users at mcs.anl.gov>
> 
> Projet_TRIOU <triou at cea.fr> writes:
> 
>> Sorry a typo, it is really MatMPISBAIJSetPreallocation()
>> 
>> Yes, I call MatMPISBAIJSetPreallocation() on ALL processes and
>> sometimes a local part of the matrix has zero rows. It worked well
>> with Petsc 3.3 and before in this particular case.
> 
> I believe this is the relevant conditional:
> 
>  if (nz >= 0 || nnz) realalloc = PETSC_TRUE;
> 
> Instead of this:
> 
>       if (nb_rows==0) // Local matrix is empty
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, NULL, 0, NULL);
>       else
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
> 
> can you just call the following?
> 
>          ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, d_nnz.addr(), 0, o_nnz.addr());
> 
> If you are passing an nnz array (can be empty if there are zero rows),
> we will trust that.  PETSC_DEFAULT is only intended to be used if you
> don't pass an array, though we're currently being tolerant by ignoring
> it if the array is non-NULL.  The problem arises because we can't tell
> the difference between a zero-length semantically-meaningful array and
> an "I'm not passing this argument" NULL.
> 
> Is this something we can fix in the documentation or should we guard in
> the code?



From jed at jedbrown.org  Thu Jan 16 10:34:15 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 16 Jan 2014 08:34:15 -0800
Subject: [petsc-dev] Fwd: [petsc-users] MatAssemblyEnd hangs during a
	parallel calculation with PETSc>3.3
In-Reply-To: <4A1F82DF-A687-4876-8028-281BDFBDDE7E@mcs.anl.gov>
References: <8738ko582c.fsf@jedbrown.org>
	<4A1F82DF-A687-4876-8028-281BDFBDDE7E@mcs.anl.gov>
Message-ID: <87wqhz51iw.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>     I think it is better to make this safe. If user calls
>     MatXXXSetPreallocation() then set nonew flag, if they do not call
>     it then do not set nonew flag. Having it depend on the exact
>     arguments passed in is just too dangerous.

That's fine, but Pierre shouldn't have to write ugly code in the mean time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140116/6fb29f05/attachment.sig>

From s.kramer at imperial.ac.uk  Fri Jan 17 05:43:46 2014
From: s.kramer at imperial.ac.uk (Stephan Kramer)
Date: Fri, 17 Jan 2014 11:43:46 +0000
Subject: [petsc-dev] bug in MatZeroRowsColumns for MPIAIJ
In-Reply-To: <CAMYG4Gn2e4-PpJ02VyRBvicmYuwFB8JgW+3A_ZqmXE4HQ8UjBA@mail.gmail.com>
References: <529A2AB2.9040004@imperial.ac.uk>
	<CAMYG4GniTfYmUKCDJf1UC_SwF6=4C0=Fd9E8QY-rz5cGRY1GJA@mail.gmail.com>
	<52BEE7C6.40609@imperial.ac.uk>
	<CAMYG4Gn2e4-PpJ02VyRBvicmYuwFB8JgW+3A_ZqmXE4HQ8UjBA@mail.gmail.com>
Message-ID: <52D91772.9060402@imperial.ac.uk>

>
> Okay, I have checked in preliminary code. It passes the tests for Mat ex18. I have merged it to next. Let me know how
> it works for you.
>
>    Thanks,
>
>       Matt
>

I've tried it out with various configurations of the test and it seems to work fine for me. I did first hit a segfault due to the uninitialised len variable, but I see that's fixed now. I've also 
tried it out in our own code (Fluidity) for what we actually want to use it for (getting rid of the ugly hack of setting bcs with big numbers on the diagonal) and it passes all tests. So thanks a lot 
for your effort: this is very useful for us.

I think it might be worth it changing the blocksize bs in the example to something bigger than 1, so we test non-trivial block-sizes? Also the nonlocalBC variable is uninitialised. I actually found a 
bug in my test if you set nonlocalBC = PETSC_TRUE with rank>3. I've pasted the diff below

Cheers
Stephan

diff --git a/src/mat/examples/tests/ex18.c b/src/mat/examples/tests/ex18.c
index 2d9ef25..40f3b9e 100644
--- a/src/mat/examples/tests/ex18.c
+++ b/src/mat/examples/tests/ex18.c
@@ -9,12 +9,12 @@ int main(int argc,char **args)
  {
    Mat            A;
    Vec            x, rhs, y;
-  PetscInt       i,j,k,b,m = 3,n,nlocal=2,bs=1,Ii,J;
+  PetscInt       i,j,k,b,m = 3,n,nlocal=2,bs=2,Ii,J;
    PetscInt       *boundary_nodes, nboundary_nodes, *boundary_indices;
    PetscMPIInt    rank,size;
    PetscErrorCode ierr;
    PetscScalar    v,v0,v1,v2,a0=0.1,a,rhsval, *boundary_values;
-  PetscBool      upwind = PETSC_FALSE, nonlocalBC;
+  PetscBool      upwind = PETSC_FALSE, nonlocalBC = PETSC_TRUE;

    ierr = PetscInitialize(&argc,&args,(char*)0,help);CHKERRQ(ierr);
    ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
@@ -80,11 +80,15 @@ int main(int argc,char **args)
        ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
        k = 0;
        for (i=size; i<m; i++,k++) {boundary_nodes[k] = n*i;};
-    } else {
-      nboundary_nodes = nlocal+1;
+    } else if (rank<m) {
+      nboundary_nodes = nlocal + 1;
        ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
        boundary_nodes[0] = rank*n;
        k = 1;
+    } else {
+      nboundary_nodes = nlocal;
+      ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
+      k = 0;
      };
      for (j=nlocal*rank; j<nlocal*(rank+1); j++,k++) {boundary_nodes[k] = j;};
    } else {


From knepley at gmail.com  Fri Jan 17 10:57:54 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 10:57:54 -0600
Subject: [petsc-dev] bug in MatZeroRowsColumns for MPIAIJ
In-Reply-To: <52D91772.9060402@imperial.ac.uk>
References: <529A2AB2.9040004@imperial.ac.uk>
	<CAMYG4GniTfYmUKCDJf1UC_SwF6=4C0=Fd9E8QY-rz5cGRY1GJA@mail.gmail.com>
	<52BEE7C6.40609@imperial.ac.uk>
	<CAMYG4Gn2e4-PpJ02VyRBvicmYuwFB8JgW+3A_ZqmXE4HQ8UjBA@mail.gmail.com>
	<52D91772.9060402@imperial.ac.uk>
Message-ID: <CAMYG4GnXZMmArDxtMhH9uk8mYUXJo5nnEfHNTEpnGSb6LkryYw@mail.gmail.com>

On Fri, Jan 17, 2014 at 5:43 AM, Stephan Kramer <s.kramer at imperial.ac.uk>wrote:

>
>> Okay, I have checked in preliminary code. It passes the tests for Mat
>> ex18. I have merged it to next. Let me know how
>> it works for you.
>>
>>    Thanks,
>>
>>       Matt
>>
>>
> I've tried it out with various configurations of the test and it seems to
> work fine for me. I did first hit a segfault due to the uninitialised len
> variable, but I see that's fixed now. I've also tried it out in our own
> code (Fluidity) for what we actually want to use it for (getting rid of the
> ugly hack of setting bcs with big numbers on the diagonal) and it passes
> all tests. So thanks a lot for your effort: this is very useful for us.
>
> I think it might be worth it changing the blocksize bs in the example to
> something bigger than 1, so we test non-trivial block-sizes? Also the
> nonlocalBC variable is uninitialised. I actually found a bug in my test if
> you set nonlocalBC = PETSC_TRUE with rank>3. I've pasted the diff below
>

I have made these fixes, added more tests, and put it in the nightly builds.

  Thanks,

     Matt


> Cheers
> Stephan
>
> diff --git a/src/mat/examples/tests/ex18.c b/src/mat/examples/tests/ex18.c
> index 2d9ef25..40f3b9e 100644
> --- a/src/mat/examples/tests/ex18.c
> +++ b/src/mat/examples/tests/ex18.c
> @@ -9,12 +9,12 @@ int main(int argc,char **args)
>  {
>    Mat            A;
>    Vec            x, rhs, y;
> -  PetscInt       i,j,k,b,m = 3,n,nlocal=2,bs=1,Ii,J;
> +  PetscInt       i,j,k,b,m = 3,n,nlocal=2,bs=2,Ii,J;
>    PetscInt       *boundary_nodes, nboundary_nodes, *boundary_indices;
>    PetscMPIInt    rank,size;
>    PetscErrorCode ierr;
>    PetscScalar    v,v0,v1,v2,a0=0.1,a,rhsval, *boundary_values;
> -  PetscBool      upwind = PETSC_FALSE, nonlocalBC;
> +  PetscBool      upwind = PETSC_FALSE, nonlocalBC = PETSC_TRUE;
>
>    ierr = PetscInitialize(&argc,&args,(char*)0,help);CHKERRQ(ierr);
>    ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
> @@ -80,11 +80,15 @@ int main(int argc,char **args)
>        ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
>        k = 0;
>        for (i=size; i<m; i++,k++) {boundary_nodes[k] = n*i;};
> -    } else {
> -      nboundary_nodes = nlocal+1;
> +    } else if (rank<m) {
> +      nboundary_nodes = nlocal + 1;
>        ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
>        boundary_nodes[0] = rank*n;
>        k = 1;
> +    } else {
> +      nboundary_nodes = nlocal;
> +      ierr = PetscMalloc1(nboundary_nodes,&boundary_nodes);CHKERRQ(ierr);
> +      k = 0;
>      };
>      for (j=nlocal*rank; j<nlocal*(rank+1); j++,k++) {boundary_nodes[k] =
> j;};
>    } else {
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/dd2d97de/attachment.html>

From jed at jedbrown.org  Fri Jan 17 11:48:05 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 09:48:05 -0800
Subject: [petsc-dev] CFLAGS and COPTFLAGS
Message-ID: <87zjmu1ove.fsf@jedbrown.org>

As far as I know, this is not a widespread convention.  Lots of users
put in CFLAGS=-O3 and either don't notice that it doesn't get used or
are frustrated that it's not used.  Is the "auto" value of splitting out
COPTFLAGS worth this confusion?  Or can we make the behavior more
obvious?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/4af1bff7/attachment.sig>

From balay at mcs.anl.gov  Fri Jan 17 11:56:09 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 17 Jan 2014 11:56:09 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87zjmu1ove.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401171155020.30127@asterix>

On Fri, 17 Jan 2014, Jed Brown wrote:

> As far as I know, this is not a widespread convention.  Lots of users
> put in CFLAGS=-O3 and either don't notice that it doesn't get used or
> are frustrated that it's not used.  Is the "auto" value of splitting out
> COPTFLAGS worth this confusion?  Or can we make the behavior more
> obvious?

This is primarily required for the option --with-debugging=0/1 [which
is also not a convention used by other packages]

Satish


From jed at jedbrown.org  Fri Jan 17 12:06:41 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 10:06:41 -0800
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <alpine.LFD.2.11.1401171155020.30127@asterix>
References: <87zjmu1ove.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401171155020.30127@asterix>
Message-ID: <87sism1o0e.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> On Fri, 17 Jan 2014, Jed Brown wrote:
>
>> As far as I know, this is not a widespread convention.  Lots of users
>> put in CFLAGS=-O3 and either don't notice that it doesn't get used or
>> are frustrated that it's not used.  Is the "auto" value of splitting out
>> COPTFLAGS worth this confusion?  Or can we make the behavior more
>> obvious?
>
> This is primarily required for the option --with-debugging=0/1 [which
> is also not a convention used by other packages]

It's not uncommon to have such a flag to enable debugging _code_ within
the library.  IIRC, it usually implies adding a debugging flag unless
the user specifies CFLAGS themselves.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/76a42624/attachment.sig>

From andrea.lani at gmail.com  Fri Jan 17 13:29:29 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Fri, 17 Jan 2014 20:29:29 +0100
Subject: [petsc-dev] GPU preconditioners
Message-ID: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>

Dear Devs,

is the PCBJACOBI solver fully ported to GPU in the latest petsc-dev
version? if not, is there any intention to do so in the near future?

I have a convection-dominated (with strong discontinuities in the flow
field) MHD problem where PCASM and PCBJACOBI both work fine with KSPGMRES.

Looking for a speedup in my code (which is using a petsc-dev version about
two-months old), the only GPU alternative I found was PCJACOBI. This is
indeed considerably faster on GPU, but does not converge (neither does,
consistently, its CPU counterpart) if not for a few iterations before
blowing up. Any other alternative for GPU-based preconditioners or solvers
worth trying at the moment?

Thanks in advance for your advice

Andrea



-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/5e11aae6/attachment.html>

From knepley at gmail.com  Fri Jan 17 13:48:14 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 13:48:14 -0600
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
Message-ID: <CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>

On Fri, Jan 17, 2014 at 1:29 PM, Andrea Lani <andrea.lani at gmail.com> wrote:

> Dear Devs,
>
> is the PCBJACOBI solver fully ported to GPU in the latest petsc-dev
> version? if not, is there any intention to do so in the near future?
>
> I have a convection-dominated (with strong discontinuities in the flow
> field) MHD problem where PCASM and PCBJACOBI both work fine with KSPGMRES.
>

These can be done on the GPU, but the key is the inner PC. Right now, we
have no ILU0 or equivalent, which I think is what
you want. You can try the AMG variants, but I am guessing they would not be
great for convection dominated flow.

Maybe Karl has a better suggestion?

  Thanks,

     Matt


> Looking for a speedup in my code (which is using a petsc-dev version about
> two-months old), the only GPU alternative I found was PCJACOBI. This is
> indeed considerably faster on GPU, but does not converge (neither does,
> consistently, its CPU counterpart) if not for a few iterations before
> blowing up. Any other alternative for GPU-based preconditioners or solvers
> worth trying at the moment?
>
> Thanks in advance for your advice
>
> Andrea
>
>
>
> --
> Dr. Andrea
> Lani
> Senior Research Engineer, PhD
> Aeronautics & Aerospace dept., CFD group
> Von Karman Institute for Fluid Dynamics
> Chausse de Waterloo 72,
> B-1640, Rhode-Saint-Genese,  Belgium
> fax  : +32-2-3599600
>  work : +32-2-3599769
> *lani at vki.ac.be <lani at vki.ac.be>*
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/8fe443ea/attachment.html>

From bsmith at mcs.anl.gov  Fri Jan 17 14:05:55 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 17 Jan 2014 14:05:55 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87zjmu1ove.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
Message-ID: <84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>


  I do not understand the context of this email at all.

  Are you referring to putting CFLAGS=-O3  inside a ./configure option? If so I see    help.addArgument('Compilers', '-CFLAGS=<string>',       nargs.Arg(None, None, 'Specify the C compiler options?))

  Or are you referring to make CFLAGS=-O3  ex1 which certainly works but I don?t recommend it?

  Is the problem that though CFLAGS=-O3 should work for configure, there is a bug in configure where it gets ignored? It seems reasonable to me that this should work.

  Barry


On Jan 17, 2014, at 11:48 AM, Jed Brown <jed at jedbrown.org> wrote:

> As far as I know, this is not a widespread convention.  Lots of users
> put in CFLAGS=-O3 and either don't notice that it doesn't get used or
> are frustrated that it's not used.  Is the "auto" value of splitting out
> COPTFLAGS worth this confusion?  Or can we make the behavior more
> obvious?



From andrea.lani at gmail.com  Fri Jan 17 14:47:31 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Fri, 17 Jan 2014 21:47:31 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
Message-ID: <8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>

Ok, thanks.

In fact, I have another major problem: when running on multi-GPU with PETSc my results are totally inconsistent compared to a single GPU  .

In my code, for now, I'm assuming a 1-1 correspondence between CPU and GPU: I run on 8 cores and 8 GPUs (4 K10).  How can I enforce this in the PETSc solver? Is it automatically done or do I have to specify some options?

Thanks again

Andrea





On Jan 17, 2014, at 8:48 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Jan 17, 2014 at 1:29 PM, Andrea Lani <andrea.lani at gmail.com> wrote:
> Dear Devs,
> 
> is the PCBJACOBI solver fully ported to GPU in the latest petsc-dev version? if not, is there any intention to do so in the near future?  
> 
> I have a convection-dominated (with strong discontinuities in the flow field) MHD problem where PCASM and PCBJACOBI both work fine with KSPGMRES.
> 
> These can be done on the GPU, but the key is the inner PC. Right now, we have no ILU0 or equivalent, which I think is what
> you want. You can try the AMG variants, but I am guessing they would not be great for convection dominated flow.
> 
> Maybe Karl has a better suggestion?
> 
>   Thanks,
> 
>      Matt
>  
> Looking for a speedup in my code (which is using a petsc-dev version about two-months old), the only GPU alternative I found was PCJACOBI. This is indeed considerably faster on GPU, but does not converge (neither does, consistently, its CPU counterpart) if not for a few iterations before blowing up. Any other alternative for GPU-based preconditioners or solvers worth trying at the moment?
> 
> Thanks in advance for your advice
> 
> Andrea
> 
> 
> 
> -- 
> Dr. Andrea Lani                                                               
> Senior Research Engineer, PhD
> Aeronautics & Aerospace dept., CFD group 
> Von Karman Institute for Fluid Dynamics   
> Chausse de Waterloo 72, 
> B-1640, Rhode-Saint-Genese,  Belgium
> fax  : +32-2-3599600  
> work : +32-2-3599769                
> lani at vki.ac.be
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/3ab651f5/attachment.html>

From andrea.lani at gmail.com  Fri Jan 17 15:00:25 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Fri, 17 Jan 2014 22:00:25 +0100
Subject: [petsc-dev] Fwd:  GPU preconditioners
In-Reply-To: <8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
Message-ID: <CALjjKKU-2vb-hmSjobJX=DFH1OYUTy9VtDJiv0vyG_57-7oJ7w@mail.gmail.com>

When I speak about totally inconsistent results between single and
multi-GPU in my previous e-mail, I mean with PCASM.  I don't see the same
with  PCJACOBI (even though, in this case, this is useless since it crashes
after a few iterations), so I'm doubting about what could be the source of
the problem.
When constructing the matrix for the multi-GPU case, I use this sequence of
calls:

MatSetType(m_mat, MATMPIAIJCUSP);
MatSetBlockSize(m_mat, blockSize);
MatMPIAIJSetPreallocation(m_mat, dnz, dnnz, onz, onnz);

and my sparsity pattern is computed accordingly, considering that I have an
AIJ matrix instead of the BAIJ I use on the "CPU version".




---------- Forwarded message ----------
From: Andrea Lani <andrea.lani at gmail.com>
Date: Fri, Jan 17, 2014 at 9:47 PM
Subject: Re: [petsc-dev] GPU preconditioners
To: Matthew Knepley <knepley at gmail.com>
Cc: For users of the development version of PETSc <petsc-dev at mcs.anl.gov>


Ok, thanks.

In fact, I have another major problem: when running on multi-GPU with PETSc
my results are totally inconsistent compared to a single GPU  .

In my code, for now, I'm assuming a 1-1 correspondence between CPU and GPU:
I run on 8 cores and 8 GPUs (4 K10).  How can I enforce this in the PETSc
solver? Is it automatically done or do I have to specify some options?

Thanks again

Andrea





On Jan 17, 2014, at 8:48 PM, Matthew Knepley <knepley at gmail.com> wrote:

On Fri, Jan 17, 2014 at 1:29 PM, Andrea Lani < <andrea.lani at gmail.com>
andrea.lani at gmail.com> wrote:

> Dear Devs,
>
> is the PCBJACOBI solver fully ported to GPU in the latest petsc-dev
> version? if not, is there any intention to do so in the near future?
>
> I have a convection-dominated (with strong discontinuities in the flow
> field) MHD problem where PCASM and PCBJACOBI both work fine with KSPGMRES.
>

These can be done on the GPU, but the key is the inner PC. Right now, we
have no ILU0 or equivalent, which I think is what
you want. You can try the AMG variants, but I am guessing they would not be
great for convection dominated flow.

Maybe Karl has a better suggestion?

  Thanks,

     Matt


> Looking for a speedup in my code (which is using a petsc-dev version about
> two-months old), the only GPU alternative I found was PCJACOBI. This is
> indeed considerably faster on GPU, but does not converge (neither does,
> consistently, its CPU counterpart) if not for a few iterations before
> blowing up. Any other alternative for GPU-based preconditioners or solvers
> worth trying at the moment?
>
> Thanks in advance for your advice
>
> Andrea
>
>
>
> --
> Dr. Andrea
> Lani
> Senior Research Engineer, PhD
> Aeronautics & Aerospace dept., CFD group
> Von Karman Institute for Fluid Dynamics
> Chausse de Waterloo 72,
> B-1640, Rhode-Saint-Genese,  Belgium
> fax  : +32-2-3599600
>  work : +32-2-3599769
> * <lani at vki.ac.be>lani at vki.ac.be <lani at vki.ac.be>*
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener




-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/69ce3a08/attachment.html>

From rupp at mcs.anl.gov  Fri Jan 17 15:02:04 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Fri, 17 Jan 2014 22:02:04 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
Message-ID: <52D99A4C.1030900@mcs.anl.gov>

Hi Andrea,

> In fact, I have another major problem: when running on multi-GPU with
> PETSc my results are totally inconsistent compared to a single GPU  .

This was a bug which was fixed a couple of days ago. It is in branch 
'next', but not yet merged to master since it has another valgrind issue 
I haven't nailed down yet.


> In my code, for now, I'm assuming a 1-1 correspondence between CPU and
> GPU: I run on 8 cores and 8 GPUs (4 K10).  How can I enforce this in the
> PETSc solver? Is it automatically done or do I have to specify some options?

One MPI rank maps to one logical GPU. In your case, please run with 8 
MPI ranks and distribute them equally over the nodes equipped with the GPUs.

As for the preconditioners: We haven't added any new preconditioners 
recently. Preconditioning on GPUs is a very problem-specific thing due 
to the burden of PCI-Express latency. Massively parallel approaches such 
as Sparse Approximate Inverses perform well in terms of theoretical FLOP 
counts, but are poor in terms of convergence and pretty expensive in 
terms of memory when running many simultaneous factorizations. ILU on 
the GPU can be fast if you order the unknowns properly and have only few 
nonzeros per row, but it is not great in terms of convergence rate 
either. PCI-Express bandwidth and latency is really a problem here...

How large are your blocks when using a block-Jacobi preconditioner for 
your problem? In the order of 3x3 or (much) larger?

Best regards,
Karli



From andrea.lani at gmail.com  Fri Jan 17 15:13:54 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Fri, 17 Jan 2014 22:13:54 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <52D99A4C.1030900@mcs.anl.gov>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
	<52D99A4C.1030900@mcs.anl.gov>
Message-ID: <CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>

Well, I have 9 equations, so 9x9 I guess...

I hope the one you are mentioning was a major bug, because what I get is
seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual of
+0.72, on 8-cores/GPU I get -1.00 at the first time step, just to make an
example. Can this be due to the bug you are saying or you can suspect
something more?

What should I do then? wait for the valgrind fix which is underway and then
update? Can you please notify me when this is fixed? I'm writing a final
report for a project and I would like to include this feature fully fixed
if possible.

Another question, what do you exactly mean by "order the unknowns properly"
in this case?
Thanks a lot!

Andrea


On Fri, Jan 17, 2014 at 10:02 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Andrea,
>
>
>  In fact, I have another major problem: when running on multi-GPU with
>> PETSc my results are totally inconsistent compared to a single GPU  .
>>
>
> This was a bug which was fixed a couple of days ago. It is in branch
> 'next', but not yet merged to master since it has another valgrind issue I
> haven't nailed down yet.
>
>
>
>  In my code, for now, I'm assuming a 1-1 correspondence between CPU and
>> GPU: I run on 8 cores and 8 GPUs (4 K10).  How can I enforce this in the
>> PETSc solver? Is it automatically done or do I have to specify some
>> options?
>>
>
> One MPI rank maps to one logical GPU. In your case, please run with 8 MPI
> ranks and distribute them equally over the nodes equipped with the GPUs.
>
> As for the preconditioners: We haven't added any new preconditioners
> recently. Preconditioning on GPUs is a very problem-specific thing due to
> the burden of PCI-Express latency. Massively parallel approaches such as
> Sparse Approximate Inverses perform well in terms of theoretical FLOP
> counts, but are poor in terms of convergence and pretty expensive in terms
> of memory when running many simultaneous factorizations. ILU on the GPU can
> be fast if you order the unknowns properly and have only few nonzeros per
> row, but it is not great in terms of convergence rate either. PCI-Express
> bandwidth and latency is really a problem here...
>
> How large are your blocks when using a block-Jacobi preconditioner for
> your problem? In the order of 3x3 or (much) larger?
>
> Best regards,
> Karli
>
>


-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/ee9bb9c3/attachment.html>

From rupp at mcs.anl.gov  Fri Jan 17 15:33:39 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Fri, 17 Jan 2014 22:33:39 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>	<52D99A4C.1030900@mcs.anl.gov>
	<CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
Message-ID: <52D9A1B3.7090203@mcs.anl.gov>

Hi Andrea,

 > Well, I have 9 equations, so 9x9 I guess...

Ok, this is just in the range where it is technically meaningful 
(register sizes) but getting challenging implementation-wise (explicit 
inversion formulas vs. Gauss with pivoting)


> I hope the one you are mentioning was a major bug, because what I get is
> seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual
> of +0.72, on 8-cores/GPU I get -1.00 at the first time step, just to
> make an example. Can this be due to the bug you are saying or you can
> suspect something more?

Yes, this was a major bug, breaking the matrix-vector product when using 
multiple MPI ranks with GPUs.


> What should I do then? wait for the valgrind fix which is underway and
> then update? Can you please notify me when this is fixed? I'm writing a
> final report for a project and I would like to include this feature
> fully fixed if possible.

I will merge the fix to master tomorrow when I'm back on my main GPU 
machine (there do not seem to be any problems in 'next' with the patch) 
and fix the valgrind complaints separately. The second issue is not 
directly related to the first, it only happens in the same module.

> Another question, what do you exactly mean by "order the unknowns
> properly" in this case?

If you build the elimination graph for the triangular factors of ILU 
preconditioners, then the ordering of the unknowns (i.e. the way you 
assign the degrees of freedoms (DOFs) on your mesh) can have a 
considerable influence on the amount of parallelism. The Cuthill-McKee 
algorithm for example is quite good for reducing the bandwidth of a 
sparse matrix, but it may also reduce the amount of parallelism for ILU0 
factors compared to e.g. a red-black ordering of the DOFs. I can send 
you a preprint if you're interested.

Best regards,
Karli



From andrea.lani at gmail.com  Fri Jan 17 15:40:22 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Fri, 17 Jan 2014 22:40:22 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <52D9A1B3.7090203@mcs.anl.gov>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
	<52D99A4C.1030900@mcs.anl.gov>
	<CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
	<52D9A1B3.7090203@mcs.anl.gov>
Message-ID: <CALjjKKXkuSa9MANiDNd9=aWqrfd8WntkpUW9CxGwfSm91iDSKA@mail.gmail.com>

Thanks Karl! Yes, please send your preprint, I'd like to learn more about
this.
Best regards

Andrea


On Fri, Jan 17, 2014 at 10:33 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Andrea,
>
>
> > Well, I have 9 equations, so 9x9 I guess...
>
> Ok, this is just in the range where it is technically meaningful (register
> sizes) but getting challenging implementation-wise (explicit inversion
> formulas vs. Gauss with pivoting)
>
>
>
>  I hope the one you are mentioning was a major bug, because what I get is
>> seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual
>> of +0.72, on 8-cores/GPU I get -1.00 at the first time step, just to
>> make an example. Can this be due to the bug you are saying or you can
>> suspect something more?
>>
>
> Yes, this was a major bug, breaking the matrix-vector product when using
> multiple MPI ranks with GPUs.
>
>
>
>  What should I do then? wait for the valgrind fix which is underway and
>> then update? Can you please notify me when this is fixed? I'm writing a
>> final report for a project and I would like to include this feature
>> fully fixed if possible.
>>
>
> I will merge the fix to master tomorrow when I'm back on my main GPU
> machine (there do not seem to be any problems in 'next' with the patch) and
> fix the valgrind complaints separately. The second issue is not directly
> related to the first, it only happens in the same module.
>
>
>  Another question, what do you exactly mean by "order the unknowns
>> properly" in this case?
>>
>
> If you build the elimination graph for the triangular factors of ILU
> preconditioners, then the ordering of the unknowns (i.e. the way you assign
> the degrees of freedoms (DOFs) on your mesh) can have a considerable
> influence on the amount of parallelism. The Cuthill-McKee algorithm for
> example is quite good for reducing the bandwidth of a sparse matrix, but it
> may also reduce the amount of parallelism for ILU0 factors compared to e.g.
> a red-black ordering of the DOFs. I can send you a preprint if you're
> interested.
>
> Best regards,
> Karli
>
>


-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/0be1a8bc/attachment.html>

From irving at naml.us  Fri Jan 17 16:06:20 2014
From: irving at naml.us (Geoffrey Irving)
Date: Fri, 17 Jan 2014 14:06:20 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions thread
Message-ID: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>

After a brief delay, I'm back to needing finite element energy
functions for optimization problems.  The original thread is

    http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-December/014161.html

That thread veered off into some more general discussions of the
PetscFE API, and I don't think came to a specific conclusion as to the
best way to add said energy functions.  In terms of API, what is
missing is a function to integrate a function over a space, where the
functions takes various field arguments and produces one or more
scalars.  The quadrature rule is important: in my case there will be
only one non-auxiliary field, and the integration function needs to
use the same quadrature rule.

I apologize if I failed to read through the previous discussion
correctly, and/or the required function has already been written.  If
not, I'm happy to mock something up, ideally with function signature
suggestions from others first.

Thanks,
Geoffrey


From irving at naml.us  Fri Jan 17 16:09:29 2014
From: irving at naml.us (Geoffrey Irving)
Date: Fri, 17 Jan 2014 14:09:29 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
Message-ID: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>

Do the test functions we integrate against to form residuals with
PetscFE come from the primal space or the dual space?  I.e., is
PetscFE always Galerkin?  If it isn't always Galerkin, when would it
be?

Thanks,
Geoffrey


From knepley at gmail.com  Fri Jan 17 17:29:47 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 17:29:47 -0600
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
Message-ID: <CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>

On Fri, Jan 17, 2014 at 4:09 PM, Geoffrey Irving <irving at naml.us> wrote:

> Do the test functions we integrate against to form residuals with
> PetscFE come from the primal space or the dual space?  I.e., is
> PetscFE always Galerkin?  If it isn't always Galerkin, when would it
> be?
>

They come from the dual space. I do not think I have made any assumptions
that prevent Petrov-Galerkin, but I have also never tried an example.

You can see the code that does tabulation here:


https://bitbucket.org/petsc/petsc/src/cdce425498f34eac2bb744f37c9fe1bd5a97b9d8/src/dm/dt/interface/dtfe.c?at=master#cl-2317

   Matt


> Thanks,
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/e50b5688/attachment.html>

From jed at jedbrown.org  Fri Jan 17 17:46:39 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 15:46:39 -0800
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
Message-ID: <878uue189s.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>   I do not understand the context of this email at all.
>
>   Are you referring to putting CFLAGS=-O3 inside a ./configure option?
>   If so I see help.addArgument('Compilers', '-CFLAGS=<string>',
>   nargs.Arg(None, None, 'Specify the C compiler options?))

If you configure with CFLAGS=-O3 --with-debugging=0, configure will test
the -O flag and ultimately use -O3 -O.  The latter takes precedence, so
the -O3 is effectively ignored.  To replace the optimization flag, one
is (currently) supposed to use COPTFLAGS=-O3 --with-debugging=0.  This
is surprising because COPTFLAGS is not a widespread convention.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/245906c3/attachment.sig>

From irving at naml.us  Fri Jan 17 17:54:15 2014
From: irving at naml.us (Geoffrey Irving)
Date: Fri, 17 Jan 2014 15:54:15 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
Message-ID: <CAJ1ofpeBSpM8yzfqT-KsjhSsSJRv0B3KoXP5Pv_9TX21KT2aiw@mail.gmail.com>

On Fri, Jan 17, 2014 at 3:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 17, 2014 at 4:09 PM, Geoffrey Irving <irving at naml.us> wrote:
>>
>> Do the test functions we integrate against to form residuals with
>> PetscFE come from the primal space or the dual space?  I.e., is
>> PetscFE always Galerkin?  If it isn't always Galerkin, when would it
>> be?
>
>
> They come from the dual space. I do not think I have made any assumptions
> that prevent Petrov-Galerkin, but I have also never tried an example.
>
> You can see the code that does tabulation here:
>
> https://bitbucket.org/petsc/petsc/src/cdce425498f34eac2bb744f37c9fe1bd5a97b9d8/src/dm/dt/interface/dtfe.c?at=master#cl-2317

Thanks.  To confirm: the default thing as set up in ex12 is Galerkin,
meaning that the primal and dual space basis functions are identical?
If that's true, how does DMDAProjectFunctionalLocal work?  As far as I
know DMDAProjectFunctionalLocal projects an analytically defined
function into the primal space, but it uses the dual space to do that
via PetscDualSpaceApply, which would imply that the dual space is
knowable in terms of the primal space.  If the dual space can be
either Galerkin (same the primal) or Petrov-Galerkin (different), I
seem to be missing something.

Geoffrey


From bsmith at mcs.anl.gov  Fri Jan 17 17:57:58 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 17 Jan 2014 17:57:58 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <878uue189s.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
Message-ID: <E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>


On Jan 17, 2014, at 5:46 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>  I do not understand the context of this email at all.
>> 
>>  Are you referring to putting CFLAGS=-O3 inside a ./configure option?
>>  If so I see help.addArgument('Compilers', '-CFLAGS=<string>',
>>  nargs.Arg(None, None, 'Specify the C compiler options?))
> 
> If you configure with CFLAGS=-O3 --with-debugging=0, configure will test
> the -O flag and ultimately use -O3 -O.  The latter takes precedence, so
> the -O3 is effectively ignored.  To replace the optimization flag, one
> is (currently) supposed to use COPTFLAGS=-O3 --with-debugging=0.  This
> is surprising because COPTFLAGS is not a widespread convention.

   The problem is that CFLAGS can have all kind of flags having nothing to do with optimization. Could we do the following?

   Look through any provided CFLAGS (FFLAGS, CXXFLAGS) if we detect something that looks like optimization then act as if COPTFLAGS was set and do not set our own optimization default flag? We can also keep support for the COPTFLAGS stuff

   We can look for -O%d , are there other things to look for?  This seems easy enough to add.

   Barry

Request-assigned: Satish,Jed Look through any provided CFLAGS  to turn off default optimization from being set




From dharmareddy84 at gmail.com  Fri Jan 17 18:00:46 2014
From: dharmareddy84 at gmail.com (Dharmendar Reddy)
Date: Fri, 17 Jan 2014 18:00:46 -0600
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
Message-ID: <CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>

Is the petscFE related functions callable from fortran ?

On Fri, Jan 17, 2014 at 5:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 17, 2014 at 4:09 PM, Geoffrey Irving <irving at naml.us> wrote:
>>
>> Do the test functions we integrate against to form residuals with
>> PetscFE come from the primal space or the dual space?  I.e., is
>> PetscFE always Galerkin?  If it isn't always Galerkin, when would it
>> be?
>
>
> They come from the dual space. I do not think I have made any assumptions
> that prevent Petrov-Galerkin, but I have also never tried an example.
>
> You can see the code that does tabulation here:
>
>
> https://bitbucket.org/petsc/petsc/src/cdce425498f34eac2bb744f37c9fe1bd5a97b9d8/src/dm/dt/interface/dtfe.c?at=master#cl-2317
>
>    Matt
>
>>
>> Thanks,
>> Geoffrey
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener



-- 
-----------------------------------------------------
Dharmendar Reddy Palle
Graduate Student
Microelectronics Research center,
University of Texas at Austin,
10100 Burnet Road, Bldg. 160
MER 2.608F, TX 78758-4445
e-mail: dharmareddy84 at gmail.com
Phone: +1-512-350-9082
United States of America.
Homepage: https://webspace.utexas.edu/~dpr342


From jed at jedbrown.org  Fri Jan 17 18:00:43 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 16:00:43 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAJ1ofpeBSpM8yzfqT-KsjhSsSJRv0B3KoXP5Pv_9TX21KT2aiw@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAJ1ofpeBSpM8yzfqT-KsjhSsSJRv0B3KoXP5Pv_9TX21KT2aiw@mail.gmail.com>
Message-ID: <874n5217mc.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:
> Thanks.  To confirm: the default thing as set up in ex12 is Galerkin,
> meaning that the primal and dual space basis functions are identical?

Yes.

> If that's true, how does DMDAProjectFunctionalLocal work?  As far as I
> know DMDAProjectFunctionalLocal projects an analytically defined
> function into the primal space, but it uses the dual space to do that
> via PetscDualSpaceApply, which would imply that the dual space is
> knowable in terms of the primal space.  If the dual space can be
> either Galerkin (same the primal) or Petrov-Galerkin (different), I
> seem to be missing something.

Most Petrov-Galerkin methods use test functions that are only
modifications of the trial functions.  SUPG is the classical example,
but DPG and others are also of this form.  These are implemented by
bringing the modification into the discrete weak form, in which case
there is only one basis tabulation.

Weighted residual methods form a larger class of methods (that includes
FD and FV), for which the primal and dual spaces need to be tabulated
separately (or specialize the method).

Is there something specific you are trying to do?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/859e37f5/attachment.sig>

From jed at jedbrown.org  Fri Jan 17 18:05:50 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 16:05:50 -0800
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
Message-ID: <871u0617dt.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    The problem is that CFLAGS can have all kind of flags having
>    nothing to do with optimization. 

Right.  Do we really need to classify and do things differently?

>    Could we do the following?
>
>    Look through any provided CFLAGS (FFLAGS, CXXFLAGS) if we detect
>    something that looks like optimization then act as if COPTFLAGS was
>    set and do not set our own optimization default flag? We can also
>    keep support for the COPTFLAGS stuff
>
>    We can look for -O%d , are there other things to look for?  This seems easy enough to add.

-ftree-vectorize, -fast, -qhot -qsimd=auto
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/30b588e9/attachment.sig>

From irving at naml.us  Fri Jan 17 18:09:41 2014
From: irving at naml.us (Geoffrey Irving)
Date: Fri, 17 Jan 2014 16:09:41 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <874n5217mc.fsf@jedbrown.org>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAJ1ofpeBSpM8yzfqT-KsjhSsSJRv0B3KoXP5Pv_9TX21KT2aiw@mail.gmail.com>
	<874n5217mc.fsf@jedbrown.org>
Message-ID: <CAJ1ofpdbGzRY_YzpUJq77kKMK8bKjUWSfEUkWPdG2EFicCEZ0w@mail.gmail.com>

On Fri, Jan 17, 2014 at 4:00 PM, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>> Thanks.  To confirm: the default thing as set up in ex12 is Galerkin,
>> meaning that the primal and dual space basis functions are identical?
>
> Yes.
>
>> If that's true, how does DMDAProjectFunctionalLocal work?  As far as I
>> know DMDAProjectFunctionalLocal projects an analytically defined
>> function into the primal space, but it uses the dual space to do that
>> via PetscDualSpaceApply, which would imply that the dual space is
>> knowable in terms of the primal space.  If the dual space can be
>> either Galerkin (same the primal) or Petrov-Galerkin (different), I
>> seem to be missing something.
>
> Most Petrov-Galerkin methods use test functions that are only
> modifications of the trial functions.  SUPG is the classical example,
> but DPG and others are also of this form.  These are implemented by
> bringing the modification into the discrete weak form, in which case
> there is only one basis tabulation.
>
> Weighted residual methods form a larger class of methods (that includes
> FD and FV), for which the primal and dual spaces need to be tabulated
> separately (or specialize the method).
>
> Is there something specific you are trying to do?

No, I want normal Galerkin, and am only trying to resolve confusion
about how the code works.  Somehow I got it into my head that because
the *Project* functions used PetscDualSpaceApply to do their work, it
meant that the dual space basis functions formed an actual inverse to
the primal basis functions.  I.e., I was thinking that if an element
of the primal space looked like

    f(x) = \sum_i w_i f_i(x)

then the dual space functions g_i satisfied

    w_i = \int_V g_i f(x)

However, I clearly hadn't thought that through, since it would make
the dual space basis functions nonlocal.  Does that mean the *Project*
functions do not produce optimal answers?

Geoffrey


From jed at jedbrown.org  Fri Jan 17 18:18:49 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 16:18:49 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAJ1ofpdbGzRY_YzpUJq77kKMK8bKjUWSfEUkWPdG2EFicCEZ0w@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAJ1ofpeBSpM8yzfqT-KsjhSsSJRv0B3KoXP5Pv_9TX21KT2aiw@mail.gmail.com>
	<874n5217mc.fsf@jedbrown.org>
	<CAJ1ofpdbGzRY_YzpUJq77kKMK8bKjUWSfEUkWPdG2EFicCEZ0w@mail.gmail.com>
Message-ID: <87txd2yweu.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:
> Does that mean the *Project* functions do not produce optimal answers?

It is a "nodal" projection rather than an L^2 projection (which requires
solving with a mass matrix).

The dual space from the Ciarlet definition typically consists of point
values at nodes, plus perhaps some averages, gradients at points, etc.
Point values are the typical case.  The dual space is used to define any
continuity between elements.

PetscFE does the change of basis from a "prime basis" (something that
can be stably evaluated) to the finite element basis defined in terms of
the dual space.  See the Brenner and Scott or the FIAT paper for further
details on this.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/b51965c9/attachment.sig>

From jed at jedbrown.org  Fri Jan 17 18:20:30 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 16:20:30 -0800
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
Message-ID: <87r486ywc1.fsf@jedbrown.org>

Dharmendar Reddy <dharmareddy84 at gmail.com> writes:

> Is the petscFE related functions callable from fortran ?

No, and it still doesn't have man pages (Matt!!!), but most of the
functions will have auto Fortran stubs as soon as the man pages are
added, and any necessary custom stubs should be added at that time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/c42379c7/attachment.sig>

From bsmith at mcs.anl.gov  Fri Jan 17 19:00:25 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 17 Jan 2014 19:00:25 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <871u0617dt.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
Message-ID: <0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>


On Jan 17, 2014, at 6:05 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>   The problem is that CFLAGS can have all kind of flags having
>>   nothing to do with optimization. 
> 
> Right.  Do we really need to classify and do things differently?

   Yes, What if they just set a single flag having nothing to do with optimization. This will turn off PETSc?s attempt to provide any kind of optimization flag (even with ?with-debugging=0). I think this is bad. *

* yes we do a crummy job of selecting optimization flags, we should do better, but we shouldn?t leave no optimization just because they passed some CFLAG.

> 
>>   Could we do the following?
>> 
>>   Look through any provided CFLAGS (FFLAGS, CXXFLAGS) if we detect
>>   something that looks like optimization then act as if COPTFLAGS was
>>   set and do not set our own optimization default flag? We can also
>>   keep support for the COPTFLAGS stuff
>> 
>>   We can look for -O%d , are there other things to look for?  This seems easy enough to add.
> 
> -ftree-vectorize, -fast, -qhot -qsimd=auto

   So if they pass -fast then we simply don?t set any optimization flags ourself? We could do that.

   Barry




From jed at jedbrown.org  Fri Jan 17 21:46:59 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 20:46:59 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
Message-ID: <87d2jqymrw.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    Yes, What if they just set a single flag having nothing to do with
>    optimization. This will turn off PETSc?s attempt to provide any
>    kind of optimization flag (even with ?with-debugging=0). I think
>    this is bad. *

Is it really that bad?  Other packages usually don't try to do anything
smart and I think there is a very real cost to second-guessing the user.
Worse, the user has to wait minutes for configure to complete before
reading the printout carefully to learn what PETSc decided to use.

I think we still want warning flags because users are unlikely to guess
decent ones.  And -fPIC needs to be automatic based on shared libs.

>>>   Could we do the following?
>>> 
>>>   Look through any provided CFLAGS (FFLAGS, CXXFLAGS) if we detect
>>>   something that looks like optimization then act as if COPTFLAGS was
>>>   set and do not set our own optimization default flag? We can also
>>>   keep support for the COPTFLAGS stuff
>>> 
>>>   We can look for -O%d , are there other things to look for?  This seems easy enough to add.
>> 
>> -ftree-vectorize, -fast, -qhot -qsimd=auto
>
>    So if they pass -fast then we simply don?t set any optimization flags ourself? We could do that.

My point was that there are *lots* of optimization flags, with new ones
every release from every vendor.  We could test for a few, but can never
be comprehensive.  Do we want to be in this business?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/b52d7d61/attachment.sig>

From knepley at gmail.com  Fri Jan 17 21:51:52 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 21:51:52 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87d2jqymrw.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
Message-ID: <CAMYG4GkE6N_QcHpDP=QCKpzcaFDi7NMEpNqRGaPJg0Ma+gWXKg@mail.gmail.com>

On Fri, Jan 17, 2014 at 9:46 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>
> >    Yes, What if they just set a single flag having nothing to do with
> >    optimization. This will turn off PETSc?s attempt to provide any
> >    kind of optimization flag (even with ?with-debugging=0). I think
> >    this is bad. *
>
> Is it really that bad?  Other packages usually don't try to do anything
> smart and I think there is a very real cost to second-guessing the user.
> Worse, the user has to wait minutes for configure to complete before
> reading the printout carefully to learn what PETSc decided to use.
>
> I think we still want warning flags because users are unlikely to guess
> decent ones.  And -fPIC needs to be automatic based on shared libs.
>
> >>>   Could we do the following?
> >>>
> >>>   Look through any provided CFLAGS (FFLAGS, CXXFLAGS) if we detect
> >>>   something that looks like optimization then act as if COPTFLAGS was
> >>>   set and do not set our own optimization default flag? We can also
> >>>   keep support for the COPTFLAGS stuff
> >>>
> >>>   We can look for -O%d , are there other things to look for?  This
> seems easy enough to add.
> >>
> >> -ftree-vectorize, -fast, -qhot -qsimd=auto
> >
> >    So if they pass -fast then we simply don?t set any optimization flags
> ourself? We could do that.
>
> My point was that there are *lots* of optimization flags, with new ones
> every release from every vendor.  We could test for a few, but can never
> be comprehensive.  Do we want to be in this business?
>

I am a fan of our current organization, which is the right mix of
flexibility and performance. I don't
think we can do much better, since we are stuck with the crap command line
interface to compilers.
I hate compilers.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/b1b74bc2/attachment.html>

From knepley at gmail.com  Fri Jan 17 21:56:23 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 21:56:23 -0600
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <87r486ywc1.fsf@jedbrown.org>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
	<87r486ywc1.fsf@jedbrown.org>
Message-ID: <CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>

On Fri, Jan 17, 2014 at 6:20 PM, Jed Brown <jed at jedbrown.org> wrote:

> Dharmendar Reddy <dharmareddy84 at gmail.com> writes:
>
> > Is the petscFE related functions callable from fortran ?
>
> No, and it still doesn't have man pages (Matt!!!), but most of the
> functions will have auto Fortran stubs as soon as the man pages are
> added, and any necessary custom stubs should be added at that time.
>

I don't put the webpages in until we decide its the right interface, or at
least close.

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/00a85c0b/attachment.html>

From jed at jedbrown.org  Fri Jan 17 21:57:06 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 20:57:06 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <CAMYG4GkE6N_QcHpDP=QCKpzcaFDi7NMEpNqRGaPJg0Ma+gWXKg@mail.gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<CAMYG4GkE6N_QcHpDP=QCKpzcaFDi7NMEpNqRGaPJg0Ma+gWXKg@mail.gmail.com>
Message-ID: <877g9yymb1.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> I am a fan of our current organization, which is the right mix of
> flexibility and performance. I don't think we can do much better,
> since we are stuck with the crap command line interface to compilers.

I brought the issue up because I noticed yet another user putting
optimization flags in CFLAGS and having it do nothing.  And I myself
recall confusion attempting to set flags and having them be ignored.
And the fact that it takes minutes to find out if your options were used
is extremely frustrating.

This is actually one point where the CMake GUI or curses interface is
helpful.  The default flags get plopped into the field and the user can
edit as they see fit without needing to second-guess the system.  Of
course non-interactive configuration can't do this except to run the
entire configure and copy out the default flags.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/156c50be/attachment.sig>

From jed at jedbrown.org  Fri Jan 17 21:57:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 20:57:58 -0700
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
	<87r486ywc1.fsf@jedbrown.org>
	<CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>
Message-ID: <874n52ym9l.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Fri, Jan 17, 2014 at 6:20 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Dharmendar Reddy <dharmareddy84 at gmail.com> writes:
>>
>> > Is the petscFE related functions callable from fortran ?
>>
>> No, and it still doesn't have man pages (Matt!!!), but most of the
>> functions will have auto Fortran stubs as soon as the man pages are
>> added, and any necessary custom stubs should be added at that time.
>>
>
> I don't put the webpages in until we decide its the right interface, or at
> least close.

Do we need

Level: experimental
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/08651c29/attachment.sig>

From knepley at gmail.com  Fri Jan 17 22:02:05 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 17 Jan 2014 22:02:05 -0600
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <874n52ym9l.fsf@jedbrown.org>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
	<87r486ywc1.fsf@jedbrown.org>
	<CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>
	<874n52ym9l.fsf@jedbrown.org>
Message-ID: <CAMYG4GmT8AbOaj4LFu0qghratVBpxXCpyqz4RvkyF=fH2gg-qA@mail.gmail.com>

On Fri, Jan 17, 2014 at 9:57 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Fri, Jan 17, 2014 at 6:20 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Dharmendar Reddy <dharmareddy84 at gmail.com> writes:
> >>
> >> > Is the petscFE related functions callable from fortran ?
> >>
> >> No, and it still doesn't have man pages (Matt!!!), but most of the
> >> functions will have auto Fortran stubs as soon as the man pages are
> >> added, and any necessary custom stubs should be added at that time.
> >>
> >
> > I don't put the webpages in until we decide its the right interface, or
> at
> > least close.
>
> Do we need
>
> Level: experimental
>

We could use 'developer'. I mean I don't want to write the for a bunch a
functions we are going to change.
I have done this for hundreds of extinct functions.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/81ac219b/attachment.html>

From jed at jedbrown.org  Fri Jan 17 22:05:03 2014
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 17 Jan 2014 21:05:03 -0700
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <CAMYG4GmT8AbOaj4LFu0qghratVBpxXCpyqz4RvkyF=fH2gg-qA@mail.gmail.com>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
	<87r486ywc1.fsf@jedbrown.org>
	<CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>
	<874n52ym9l.fsf@jedbrown.org>
	<CAMYG4GmT8AbOaj4LFu0qghratVBpxXCpyqz4RvkyF=fH2gg-qA@mail.gmail.com>
Message-ID: <871u06ylxs.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> We could use 'developer'. I mean I don't want to write the for a bunch a
> functions we are going to change.
> I have done this for hundreds of extinct functions.

Sure, the problem is that we forget.  I think we should write stubs and
a tag of some sort so we know what needs to be revisited.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140117/80571c48/attachment.sig>

From rupp at mcs.anl.gov  Sat Jan 18 03:26:52 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sat, 18 Jan 2014 10:26:52 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>	<52D99A4C.1030900@mcs.anl.gov>
	<CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
Message-ID: <52DA48DC.6070604@mcs.anl.gov>

Hi Andrea,

the fix is now merged to master:
https://bitbucket.org/petsc/petsc/commits/087a195f1d07b315894e9d8ab1801a0ce993221c

Best regards,
Karli



On 01/17/2014 10:13 PM, Andrea Lani wrote:
> Well, I have 9 equations, so 9x9 I guess...
>
> I hope the one you are mentioning was a major bug, because what I get is
> seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual
> of +0.72, on 8-cores/GPU I get -1.00 at the first time step, just to
> make an example. Can this be due to the bug you are saying or you can
> suspect something more?
>
> What should I do then? wait for the valgrind fix which is underway and
> then update? Can you please notify me when this is fixed? I'm writing a
> final report for a project and I would like to include this feature
> fully fixed if possible.
>
> Another question, what do you exactly mean by "order the unknowns
> properly" in this case?
> Thanks a lot!
>
> Andrea
>
>
> On Fri, Jan 17, 2014 at 10:02 PM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Andrea,
>
>
>         In fact, I have another major problem: when running on multi-GPU
>         with
>         PETSc my results are totally inconsistent compared to a single
>         GPU  .
>
>
>     This was a bug which was fixed a couple of days ago. It is in branch
>     'next', but not yet merged to master since it has another valgrind
>     issue I haven't nailed down yet.
>
>
>
>         In my code, for now, I'm assuming a 1-1 correspondence between
>         CPU and
>         GPU: I run on 8 cores and 8 GPUs (4 K10).  How can I enforce
>         this in the
>         PETSc solver? Is it automatically done or do I have to specify
>         some options?
>
>
>     One MPI rank maps to one logical GPU. In your case, please run with
>     8 MPI ranks and distribute them equally over the nodes equipped with
>     the GPUs.
>
>     As for the preconditioners: We haven't added any new preconditioners
>     recently. Preconditioning on GPUs is a very problem-specific thing
>     due to the burden of PCI-Express latency. Massively parallel
>     approaches such as Sparse Approximate Inverses perform well in terms
>     of theoretical FLOP counts, but are poor in terms of convergence and
>     pretty expensive in terms of memory when running many simultaneous
>     factorizations. ILU on the GPU can be fast if you order the unknowns
>     properly and have only few nonzeros per row, but it is not great in
>     terms of convergence rate either. PCI-Express bandwidth and latency
>     is really a problem here...
>
>     How large are your blocks when using a block-Jacobi preconditioner
>     for your problem? In the order of 3x3 or (much) larger?
>
>     Best regards,
>     Karli
>
>
>
>
> --
> Dr. Andrea Lani
> Senior Research Engineer, PhD
> Aeronautics & Aerospace dept., CFD group
> Von Karman Institute for Fluid Dynamics
> Chausse de Waterloo 72,
> B-1640, Rhode-Saint-Genese,  Belgium
> fax  : +32-2-3599600
> work : +32-2-3599769 _
> lani at vki.ac.be <mailto:lani at vki.ac.be>_



From andrea.lani at gmail.com  Sat Jan 18 03:53:02 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Sat, 18 Jan 2014 10:53:02 +0100
Subject: [petsc-dev] GPU preconditioners
In-Reply-To: <52DA48DC.6070604@mcs.anl.gov>
References: <CALjjKKWRYY0MnVfosXW55ND0SyRsF30ThfVO__krcaijNuyC6w@mail.gmail.com>
	<CAMYG4GmLLeKv3Kx0GLZ+xZ4Z=n1kaBFrBuOhk2fktVva7F7dRA@mail.gmail.com>
	<8E1B66E7-593B-462F-BCD5-4DFBB1398EAC@gmail.com>
	<52D99A4C.1030900@mcs.anl.gov>
	<CALjjKKW30JphUmj694M3q4OuKVj=JAf=AgWN0PiYmftBpd61jw@mail.gmail.com>
	<52DA48DC.6070604@mcs.anl.gov>
Message-ID: <EE8086E4-2B1E-4AE7-8512-97DD21726BBA@gmail.com>

Thanks a lot, Karli! I will update, make a few tests and let you know if my problem is fixed!
Best regards

Andrea

On Jan 18, 2014, at 10:26 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Andrea,
> 
> the fix is now merged to master:
> https://bitbucket.org/petsc/petsc/commits/087a195f1d07b315894e9d8ab1801a0ce993221c
> 
> Best regards,
> Karli
> 
> 
> 
> On 01/17/2014 10:13 PM, Andrea Lani wrote:
>> Well, I have 9 equations, so 9x9 I guess...
>> 
>> I hope the one you are mentioning was a major bug, because what I get is
>> seriously wrong: while on single GPU (KSPGMRES+PCASM) I get a residual
>> of +0.72, on 8-cores/GPU I get -1.00 at the first time step, just to
>> make an example. Can this be due to the bug you are saying or you can
>> suspect something more?
>> 
>> What should I do then? wait for the valgrind fix which is underway and
>> then update? Can you please notify me when this is fixed? I'm writing a
>> final report for a project and I would like to include this feature
>> fully fixed if possible.
>> 
>> Another question, what do you exactly mean by "order the unknowns
>> properly" in this case?
>> Thanks a lot!
>> 
>> Andrea
>> 
>> 
>> On Fri, Jan 17, 2014 at 10:02 PM, Karl Rupp <rupp at mcs.anl.gov
>> <mailto:rupp at mcs.anl.gov>> wrote:
>> 
>>    Hi Andrea,
>> 
>> 
>>        In fact, I have another major problem: when running on multi-GPU
>>        with
>>        PETSc my results are totally inconsistent compared to a single
>>        GPU  .
>> 
>> 
>>    This was a bug which was fixed a couple of days ago. It is in branch
>>    'next', but not yet merged to master since it has another valgrind
>>    issue I haven't nailed down yet.
>> 
>> 
>> 
>>        In my code, for now, I'm assuming a 1-1 correspondence between
>>        CPU and
>>        GPU: I run on 8 cores and 8 GPUs (4 K10).  How can I enforce
>>        this in the
>>        PETSc solver? Is it automatically done or do I have to specify
>>        some options?
>> 
>> 
>>    One MPI rank maps to one logical GPU. In your case, please run with
>>    8 MPI ranks and distribute them equally over the nodes equipped with
>>    the GPUs.
>> 
>>    As for the preconditioners: We haven't added any new preconditioners
>>    recently. Preconditioning on GPUs is a very problem-specific thing
>>    due to the burden of PCI-Express latency. Massively parallel
>>    approaches such as Sparse Approximate Inverses perform well in terms
>>    of theoretical FLOP counts, but are poor in terms of convergence and
>>    pretty expensive in terms of memory when running many simultaneous
>>    factorizations. ILU on the GPU can be fast if you order the unknowns
>>    properly and have only few nonzeros per row, but it is not great in
>>    terms of convergence rate either. PCI-Express bandwidth and latency
>>    is really a problem here...
>> 
>>    How large are your blocks when using a block-Jacobi preconditioner
>>    for your problem? In the order of 3x3 or (much) larger?
>> 
>>    Best regards,
>>    Karli
>> 
>> 
>> 
>> 
>> --
>> Dr. Andrea Lani
>> Senior Research Engineer, PhD
>> Aeronautics & Aerospace dept., CFD group
>> Von Karman Institute for Fluid Dynamics
>> Chausse de Waterloo 72,
>> B-1640, Rhode-Saint-Genese,  Belgium
>> fax  : +32-2-3599600
>> work : +32-2-3599769 _
>> lani at vki.ac.be <mailto:lani at vki.ac.be>_
> 


From knepley at gmail.com  Sat Jan 18 09:03:56 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 18 Jan 2014 09:03:56 -0600
Subject: [petsc-dev] are petscfe tests functions dual or primal?
In-Reply-To: <871u06ylxs.fsf@jedbrown.org>
References: <CAJ1ofpeGdkPJfn_9E_A8NUrv8sZxJH4nB-=JD7CkwX9oLFk+0g@mail.gmail.com>
	<CAMYG4Gm2Uf5G2mztN0En4KgNQYNiAFZY4YN8Z69VGHCbhb4pSQ@mail.gmail.com>
	<CAOwTPdcQ6HkiAtRv-8VeN2B01Buycu9CWBtcF3hJixZMv3iF_w@mail.gmail.com>
	<87r486ywc1.fsf@jedbrown.org>
	<CAMYG4Gm8wGYCsRgJTagsGDOEYAi2iB+jCo6dcjQiZCniJP2HcQ@mail.gmail.com>
	<874n52ym9l.fsf@jedbrown.org>
	<CAMYG4GmT8AbOaj4LFu0qghratVBpxXCpyqz4RvkyF=fH2gg-qA@mail.gmail.com>
	<871u06ylxs.fsf@jedbrown.org>
Message-ID: <CAMYG4G=Y19U9wG_j-rpzYeX=BD=qw3qjmJn9ozgOjJJtKpLZCw@mail.gmail.com>

On Fri, Jan 17, 2014 at 10:05 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > We could use 'developer'. I mean I don't want to write the for a bunch a
> > functions we are going to change.
> > I have done this for hundreds of extinct functions.
>
> Sure, the problem is that we forget.  I think we should write stubs and
> a tag of some sort so we know what needs to be revisited.
>

I am fine with that.

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/df316971/attachment.html>

From bsmith at mcs.anl.gov  Sat Jan 18 10:21:48 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 18 Jan 2014 10:21:48 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87d2jqymrw.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
Message-ID: <0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>


On Jan 17, 2014, at 9:46 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>   
>>>> 
>>>>  We can look for -O%d , are there other things to look for?  This seems easy enough to add.
>>> 
>>> -ftree-vectorize, -fast, -qhot -qsimd=auto
>> 
>>   So if they pass -fast then we simply don?t set any optimization flags ourself? We could do that.
> 
> My point was that there are *lots* of optimization flags, with new ones
> every release from every vendor.  We could test for a few, but can never
> be comprehensive.  Do we want to be in this business?

   Do people really set some of the above and not set the -O%d one at the same time?

   My plan was if CFLAGS contained a -O[%d] then we turn off automatic setting of optimization flags. Why would this not work?

   I agree with you we currently have a problem but I want a solution better than the community.

   Barry




From jed at jedbrown.org  Sat Jan 18 10:43:27 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sat, 18 Jan 2014 09:43:27 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
Message-ID: <874n51xmts.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    Do people really set some of the above and not set the -O%d one at the same time?

I don't know.  Sometimes the MPI wrapper adds the -O flag itself.  The
user might know that and thus not bother adding it themselves.  I think
it would be surprising for some special flags to cause such changes in
behavior.  I don't have a strong opinion on this point, but I think we
should do something better than we have now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/7e257552/attachment.sig>

From wgropp at illinois.edu  Sat Jan 18 10:49:07 2014
From: wgropp at illinois.edu (William Gropp)
Date: Sat, 18 Jan 2014 10:49:07 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <874n51xmts.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
Message-ID: <7BA562DB-1ABC-41F9-ABAC-EF61BA917B19@illinois.edu>

If the MPI wrapper adds -O, its buggy.  The wrapper should only do what is necessary to get MPI to compile (header and library paths) and nothing else.  If MPICH does more, that is a bug (note that recent autoconf really, really wants to specify -g -O2 for gcc; that doesn't change the fact that it is wrong to do so).

Bill

William Gropp
Director, Parallel Computing Institute
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign




On Jan 18, 2014, at 10:43 AM, Jed Brown wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>   Do people really set some of the above and not set the -O%d one at the same time?
> 
> I don't know.  Sometimes the MPI wrapper adds the -O flag itself.  The
> user might know that and thus not bother adding it themselves.  I think
> it would be surprising for some special flags to cause such changes in
> behavior.  I don't have a strong opinion on this point, but I think we
> should do something better than we have now.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/0fa9189f/attachment.html>

From bsmith at mcs.anl.gov  Sat Jan 18 11:09:15 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 18 Jan 2014 11:09:15 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <874n51xmts.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
Message-ID: <DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>


On Jan 18, 2014, at 10:43 AM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>   Do people really set some of the above and not set the -O%d one at the same time?
> 
> I don't know.  Sometimes the MPI wrapper adds the -O flag itself.  The
> user might know that and thus not bother adding it themselves.  I think
> it would be surprising for some special flags to cause such changes in
> behavior.  I don't have a strong opinion on this point, but I think we
> should do something better than we have now.

   Summary

   The reason for the current model is that in the original plan ?with-debugging=0 would providing good compiler optimization flags (for each system) without the user having to set them; if the user set them via COPTFLAGS then we did set them ourselves.

   Observations

    1)   The ?standard? for passing optimization flags to ./configure is CFLAGS=?-O3 crap? we don?t support this and this confuses users

    2) If we followed the standard and the user only set non-optimization flags with CFLAGS we would not turn on optimization when user might expect it (does this ever happen?)

    3) We provide a different, ?better" way of providing optimization flags and other CFLAGS, but know one knows about it.

    4) Our ?with-debugging=0 optimization flags are pretty bad, we should improve them

    So choices are

     A) follow standard

     B) Continue our way

      C) Hybrid that does not use our own flags if user provides optimization flag.

Request-assigned: Satish,  if CLAGS contains a -O    CXXFLAGS contains a -O FFLAGS contains a -O then turn off PETSc generation of optimization flags for that compiler

Request-assigned: Satish,  if mpicc, mpicxx, mpifc provide -O flags then turn off PETSc generation of optimization flags

    Won?t completely satisfy anyone but me.

   Barry






From jed at jedbrown.org  Sat Jan 18 11:41:29 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sat, 18 Jan 2014 10:41:29 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <7BA562DB-1ABC-41F9-ABAC-EF61BA917B19@illinois.edu>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<7BA562DB-1ABC-41F9-ABAC-EF61BA917B19@illinois.edu>
Message-ID: <871u05xk52.fsf@jedbrown.org>

William Gropp <wgropp at illinois.edu> writes:

> If the MPI wrapper adds -O, its buggy.  

MPICH puts CFLAGS into the wrappers.  Since this is the standard way of
specifying optimization when building a package, many users have
wrappers configured this way.  MPICH must be configured using
MPICHLIB_CFLAGS (formerly MPICH2LIB_CFLAGS) if they do not want the
optimization to go into the wrappers.  This causes real confusion such
as in this thread.

http://www.open-mpi.org/community/lists/users/2008/10/6924.php
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/eb2b0902/attachment.sig>

From bsmith at mcs.anl.gov  Sat Jan 18 13:19:33 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 18 Jan 2014 13:19:33 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
Message-ID: <3BFC19AF-AF8E-4941-84DF-6CF29B9D5C31@mcs.anl.gov>


   If mpicc is providing a -On flag and the PETSc ./configure user provides a -Om flag should we generate a very useful error message telling them of the conflict? 

   Even generate an error?

   Barry


On Jan 18, 2014, at 11:09 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
> On Jan 18, 2014, at 10:43 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>  Do people really set some of the above and not set the -O%d one at the same time?
>> 
>> I don't know.  Sometimes the MPI wrapper adds the -O flag itself.  The
>> user might know that and thus not bother adding it themselves.  I think
>> it would be surprising for some special flags to cause such changes in
>> behavior.  I don't have a strong opinion on this point, but I think we
>> should do something better than we have now.
> 
>   Summary
> 
>   The reason for the current model is that in the original plan ?with-debugging=0 would providing good compiler optimization flags (for each system) without the user having to set them; if the user set them via COPTFLAGS then we did set them ourselves.
> 
>   Observations
> 
>    1)   The ?standard? for passing optimization flags to ./configure is CFLAGS=?-O3 crap? we don?t support this and this confuses users
> 
>    2) If we followed the standard and the user only set non-optimization flags with CFLAGS we would not turn on optimization when user might expect it (does this ever happen?)
> 
>    3) We provide a different, ?better" way of providing optimization flags and other CFLAGS, but know one knows about it.
> 
>    4) Our ?with-debugging=0 optimization flags are pretty bad, we should improve them
> 
>    So choices are
> 
>     A) follow standard
> 
>     B) Continue our way
> 
>      C) Hybrid that does not use our own flags if user provides optimization flag.
> 
> Request-assigned: Satish,  if CLAGS contains a -O    CXXFLAGS contains a -O FFLAGS contains a -O then turn off PETSc generation of optimization flags for that compiler
> 
> Request-assigned: Satish,  if mpicc, mpicxx, mpifc provide -O flags then turn off PETSc generation of optimization flags
> 
>    Won?t completely satisfy anyone but me.
> 
>   Barry
> 
> 
> 
> 



From bsmith at mcs.anl.gov  Sat Jan 18 18:09:05 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 18 Jan 2014 18:09:05 -0600
Subject: [petsc-dev] nonlinear solvers
In-Reply-To: <CAOwTPdfRUFg7Jb_DUoo2GHNmtcOWA23gHwD5XTA86k_rdhg-ng@mail.gmail.com>
References: <CAOwTPdeJWiDqRd8wN--2UnfnMaCRfdf3Q+q0W7wcJvT7tca_hA@mail.gmail.com>
	<52DAF829.4030009@mcs.anl.gov>
	<CAOwTPdfRUFg7Jb_DUoo2GHNmtcOWA23gHwD5XTA86k_rdhg-ng@mail.gmail.com>
Message-ID: <ACF85AAB-3B8E-40C4-AE37-5DEF8E0C6F73@mcs.anl.gov>


  Peter,

   What the hay is this R. E. Bank and D. J. Rose, ``Global Approximate Newton Methods,'' Numer. Math., vol. 37, pp. 279-295, 1981. and should it be a ?line search? or something in SNES?

   Thanks

   Barry


On Jan 18, 2014, at 5:59 PM, Dharmendar Reddy <dharmareddy84 at gmail.com> wrote:

> On Sat, Jan 18, 2014 at 3:54 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:
>> Hi,
>> 
>> 
>> 
>>>                  I am solving a set of equations with SNES
>>> 
>>> F1 (x1,x2,x3) = 0
>>> F2 (x1,x2,x3) = 0
>>> F3 (x1,x2,x3) = 0
>>> 
>>> The system of equations is shown on page 1 of pdf here
>>> http://dunham.ee.washington.edu/ee531/notes/SemiRev.pdf
>>> 
>>> F1 = equation 1
>>> F2 = equation 2
>>> F3 = equation 5
>>> 
>>> x1 = n, X2=p and X3 = psi,
>>> X1 and X2 have an exponential dependance on X1
>>> after i scale the variables, X3 typically varies between say +/- 100
>>> where as X1 and X2 vary between 0 to 2. norm(X) then may usually
>>> dominated by solution values of X3.
>> 
>> 
>> If you are solving the drift-diffusion system for semiconductors, which
>> discretization do you use? How did you stabilize the strong advection?
>> 
>> 
> 
> My plan is to use the discretization method described here.
> (http://www.iue.tuwien.ac.at/phd/triebl/node30.html ).
> 
> The method typically used for for stabilizing the advection term is
> called Scharfetter-Gummel method described the above link.
> 
> When i intially started the code design, i wanted to implement the
> approach mentioned  in this paper (dx.doi.org/10.2172/1020517) . I am
> still learning about this things so..i am not sure which is the right
> way to go.
> 
> For stabilizing, i implemented the bank n rose algorithm via
> SNESPostCheck, i am yet to test the efficacy of this method over the
> defualt snes methods.
> (http://www.iue.tuwien.ac.at/phd/ceric/node17.html)
> 
> 
> 
>> 
>>> Can you suggest me the snes options that i need to use to achieve the
>>> following:
>>> 
>>> 1. X1 > 0 and X2 > 0  (as per previous emails, i can use
>>> SNESSetVariableBounds)
>> 
>> 
>> Have you considered a transformation to quasi-fermi potentials, i.e.
>> n ~ exp(phi_n), p ~ exp(phi_p)
>> or Slotboom variables? This way you could get rid of the constraint
>> entirely. Even if you solve for (n,p,psi), my experience is that positivity
>> is preserved automatically when using a good discretization and step size
>> control.
>> 
>> 
>> 
>>> 2. I want the updates to solution to have an adaptive stepping based
>>> on norm of (F) or norm(X). If norm(F) is decreasing as the iteration
>>> progresss, larger stepping others wise reduce the step size..
>>> Similarly for Norm of X.
>> 
>> 
>> A good damping for the drift-diffusion system is tricky. I know a couple of
>> empirical expressions, but would be interested to know whether this can be
>> handled in a more black-box manner as well.
>> 
>> Best regards,
>> Karli
>> 



From prbrune at gmail.com  Sat Jan 18 18:54:51 2014
From: prbrune at gmail.com (Peter Brune)
Date: Sat, 18 Jan 2014 18:54:51 -0600
Subject: [petsc-dev] nonlinear solvers
In-Reply-To: <ACF85AAB-3B8E-40C4-AE37-5DEF8E0C6F73@mcs.anl.gov>
References: <CAOwTPdeJWiDqRd8wN--2UnfnMaCRfdf3Q+q0W7wcJvT7tca_hA@mail.gmail.com>
	<52DAF829.4030009@mcs.anl.gov>
	<CAOwTPdfRUFg7Jb_DUoo2GHNmtcOWA23gHwD5XTA86k_rdhg-ng@mail.gmail.com>
	<ACF85AAB-3B8E-40C4-AE37-5DEF8E0C6F73@mcs.anl.gov>
Message-ID: <CAKKtykyvyZdtqH_D3-=ZS+Pq98ZHm_TEBG4XwOSWMxJhQFEA_g@mail.gmail.com>

>From my initial reading of the paper, it's about a backtracking linesearch
where you have a persistent-between-iterations parameter that increases or
decreases, taking your damping less than or towards one based upon the
difference between subsequent residual norms.   I have no idea how it fits
in as a post-check rather than as a line search, as it would be redundant.
 It's not clear that this would be better than anything we have now.  Are
you sure that this is the right paper?

- Peter

On Sat, Jan 18, 2014 at 6:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Peter,
>
>    What the hay is this R. E. Bank and D. J. Rose, ``Global Approximate
> Newton Methods,'' Numer. Math., vol. 37, pp. 279-295, 1981. and should it
> be a ?line search? or something in SNES?
>
>    Thanks
>
>    Barry
>
>
> On Jan 18, 2014, at 5:59 PM, Dharmendar Reddy <dharmareddy84 at gmail.com>
> wrote:
>
> > On Sat, Jan 18, 2014 at 3:54 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:
> >> Hi,
> >>
> >>
> >>
> >>>                  I am solving a set of equations with SNES
> >>>
> >>> F1 (x1,x2,x3) = 0
> >>> F2 (x1,x2,x3) = 0
> >>> F3 (x1,x2,x3) = 0
> >>>
> >>> The system of equations is shown on page 1 of pdf here
> >>> http://dunham.ee.washington.edu/ee531/notes/SemiRev.pdf
> >>>
> >>> F1 = equation 1
> >>> F2 = equation 2
> >>> F3 = equation 5
> >>>
> >>> x1 = n, X2=p and X3 = psi,
> >>> X1 and X2 have an exponential dependance on X1
> >>> after i scale the variables, X3 typically varies between say +/- 100
> >>> where as X1 and X2 vary between 0 to 2. norm(X) then may usually
> >>> dominated by solution values of X3.
> >>
> >>
> >> If you are solving the drift-diffusion system for semiconductors, which
> >> discretization do you use? How did you stabilize the strong advection?
> >>
> >>
> >
> > My plan is to use the discretization method described here.
> > (http://www.iue.tuwien.ac.at/phd/triebl/node30.html ).
> >
> > The method typically used for for stabilizing the advection term is
> > called Scharfetter-Gummel method described the above link.
> >
> > When i intially started the code design, i wanted to implement the
> > approach mentioned  in this paper (dx.doi.org/10.2172/1020517) . I am
> > still learning about this things so..i am not sure which is the right
> > way to go.
> >
> > For stabilizing, i implemented the bank n rose algorithm via
> > SNESPostCheck, i am yet to test the efficacy of this method over the
> > defualt snes methods.
> > (http://www.iue.tuwien.ac.at/phd/ceric/node17.html)
> >
> >
> >
> >>
> >>> Can you suggest me the snes options that i need to use to achieve the
> >>> following:
> >>>
> >>> 1. X1 > 0 and X2 > 0  (as per previous emails, i can use
> >>> SNESSetVariableBounds)
> >>
> >>
> >> Have you considered a transformation to quasi-fermi potentials, i.e.
> >> n ~ exp(phi_n), p ~ exp(phi_p)
> >> or Slotboom variables? This way you could get rid of the constraint
> >> entirely. Even if you solve for (n,p,psi), my experience is that
> positivity
> >> is preserved automatically when using a good discretization and step
> size
> >> control.
> >>
> >>
> >>
> >>> 2. I want the updates to solution to have an adaptive stepping based
> >>> on norm of (F) or norm(X). If norm(F) is decreasing as the iteration
> >>> progresss, larger stepping others wise reduce the step size..
> >>> Similarly for Norm of X.
> >>
> >>
> >> A good damping for the drift-diffusion system is tricky. I know a
> couple of
> >> empirical expressions, but would be interested to know whether this can
> be
> >> handled in a more black-box manner as well.
> >>
> >> Best regards,
> >> Karli
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/7268a1f9/attachment.html>

From bsmith at mcs.anl.gov  Sat Jan 18 19:06:45 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 18 Jan 2014 19:06:45 -0600
Subject: [petsc-dev] nonlinear solvers
In-Reply-To: <CAKKtykyvyZdtqH_D3-=ZS+Pq98ZHm_TEBG4XwOSWMxJhQFEA_g@mail.gmail.com>
References: <CAOwTPdeJWiDqRd8wN--2UnfnMaCRfdf3Q+q0W7wcJvT7tca_hA@mail.gmail.com>
	<52DAF829.4030009@mcs.anl.gov>
	<CAOwTPdfRUFg7Jb_DUoo2GHNmtcOWA23gHwD5XTA86k_rdhg-ng@mail.gmail.com>
	<ACF85AAB-3B8E-40C4-AE37-5DEF8E0C6F73@mcs.anl.gov>
	<CAKKtykyvyZdtqH_D3-=ZS+Pq98ZHm_TEBG4XwOSWMxJhQFEA_g@mail.gmail.com>
Message-ID: <564EF7D2-F32E-4694-B74F-81A6C6B22418@mcs.anl.gov>


On Jan 18, 2014, at 6:54 PM, Peter Brune <prbrune at gmail.com> wrote:

> From my initial reading of the paper, it's about a backtracking linesearch where you have a persistent-between-iterations parameter that increases or decreases, taking your damping less than or towards one based upon the difference between subsequent residual norms.

> 

    Thanks

     It is an example of a line search with persistent state that we do not currently have. Right?

>   I have no idea how it fits in as a post-check rather than as a line search, as it would be redundant.

   Agreed, it doesn?t belong as a post-check


>  It's not clear that this would be better than anything we have now.

    Agreed. But these are famous people so maybe there is some great subtly we are missing, or maybe not.

>  Are you sure that this is the right paper?

   Pretty sure. It is the in the link from the other paper.

   Barry

> 
> - Peter
> 
> On Sat, Jan 18, 2014 at 6:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Peter,
> 
>    What the hay is this R. E. Bank and D. J. Rose, ``Global Approximate Newton Methods,'' Numer. Math., vol. 37, pp. 279-295, 1981. and should it be a ?line search? or something in SNES?
> 
>    Thanks
> 
>    Barry
> 
> 
> On Jan 18, 2014, at 5:59 PM, Dharmendar Reddy <dharmareddy84 at gmail.com> wrote:
> 
> > On Sat, Jan 18, 2014 at 3:54 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:
> >> Hi,
> >>
> >>
> >>
> >>>                  I am solving a set of equations with SNES
> >>>
> >>> F1 (x1,x2,x3) = 0
> >>> F2 (x1,x2,x3) = 0
> >>> F3 (x1,x2,x3) = 0
> >>>
> >>> The system of equations is shown on page 1 of pdf here
> >>> http://dunham.ee.washington.edu/ee531/notes/SemiRev.pdf
> >>>
> >>> F1 = equation 1
> >>> F2 = equation 2
> >>> F3 = equation 5
> >>>
> >>> x1 = n, X2=p and X3 = psi,
> >>> X1 and X2 have an exponential dependance on X1
> >>> after i scale the variables, X3 typically varies between say +/- 100
> >>> where as X1 and X2 vary between 0 to 2. norm(X) then may usually
> >>> dominated by solution values of X3.
> >>
> >>
> >> If you are solving the drift-diffusion system for semiconductors, which
> >> discretization do you use? How did you stabilize the strong advection?
> >>
> >>
> >
> > My plan is to use the discretization method described here.
> > (http://www.iue.tuwien.ac.at/phd/triebl/node30.html ).
> >
> > The method typically used for for stabilizing the advection term is
> > called Scharfetter-Gummel method described the above link.
> >
> > When i intially started the code design, i wanted to implement the
> > approach mentioned  in this paper (dx.doi.org/10.2172/1020517) . I am
> > still learning about this things so..i am not sure which is the right
> > way to go.
> >
> > For stabilizing, i implemented the bank n rose algorithm via
> > SNESPostCheck, i am yet to test the efficacy of this method over the
> > defualt snes methods.
> > (http://www.iue.tuwien.ac.at/phd/ceric/node17.html)
> >
> >
> >
> >>
> >>> Can you suggest me the snes options that i need to use to achieve the
> >>> following:
> >>>
> >>> 1. X1 > 0 and X2 > 0  (as per previous emails, i can use
> >>> SNESSetVariableBounds)
> >>
> >>
> >> Have you considered a transformation to quasi-fermi potentials, i.e.
> >> n ~ exp(phi_n), p ~ exp(phi_p)
> >> or Slotboom variables? This way you could get rid of the constraint
> >> entirely. Even if you solve for (n,p,psi), my experience is that positivity
> >> is preserved automatically when using a good discretization and step size
> >> control.
> >>
> >>
> >>
> >>> 2. I want the updates to solution to have an adaptive stepping based
> >>> on norm of (F) or norm(X). If norm(F) is decreasing as the iteration
> >>> progresss, larger stepping others wise reduce the step size..
> >>> Similarly for Norm of X.
> >>
> >>
> >> A good damping for the drift-diffusion system is tricky. I know a couple of
> >> empirical expressions, but would be interested to know whether this can be
> >> handled in a more black-box manner as well.
> >>
> >> Best regards,
> >> Karli
> >>
> 
> 



From hzhang at mcs.anl.gov  Sat Jan 18 22:22:19 2014
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Sat, 18 Jan 2014 22:22:19 -0600
Subject: [petsc-dev] Fwd: [MCS Staff] Jed Brown wins the SIAG/Supercomputing
 Junior Scientist Prize
In-Reply-To: <a7c5abda06574a9381b815c6d80d39a5@LUCKMAN.anl.gov>
References: <a7c5abda06574a9381b815c6d80d39a5@LUCKMAN.anl.gov>
Message-ID: <CAGCphBvBUd-Y0e9giMTka1QMkHeyozf21bF3yNGjc05bUiVgSg@mail.gmail.com>

Jed,
Congratulates!!!
Hong

---------- Forwarded message ----------
From: Snir, Marc <snir at anl.gov>
Date: Sat, Jan 18, 2014 at 3:12 PM
Subject: [MCS Staff] Jed Brown wins the SIAG/Supercomputing Junior
Scientist Prize
To: "mcs at mcs.anl.gov" <mcs at mcs.anl.gov>


Please join me in congratulating Jed Brown for winning the
SIAG/Supercomputing Junior Scientist Prize

The SIAM Activity Group on Supercomputing (SIAG/SC) Junior Scientist Prize,
established in 2009, is awarded to an outstanding junior researcher in the
field of algorithms research and development for parallel scientific and
engineering computing, for distinguished contributions to the field in the
three calendar years prior to the year of the award. The prize will be
awarded next month at the SIAM Conference on Parallel Processing for
Scientific Computing (PP14).

Thanks Jed, for your work and for the added recognition to MCS.



Marc Snir

Director, Mathematics and Computer Science Division,
Argonne National Laboratory

Michael Faiman and Saburo Muroga Professor,
Department of Computer Science, University of Illinois at Urbana Champaign



[Sent to announce at mcs.anl.gov.]

_______________________________________________
Subscription to this list is automatic based on your appointment in the MCS
division.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140118/48255b99/attachment.html>

From rupp at mcs.anl.gov  Sun Jan 19 04:36:55 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sun, 19 Jan 2014 11:36:55 +0100
Subject: [petsc-dev] Fwd: [MCS Staff] Jed Brown wins the
 SIAG/Supercomputing Junior Scientist Prize
In-Reply-To: <CAGCphBvBUd-Y0e9giMTka1QMkHeyozf21bF3yNGjc05bUiVgSg@mail.gmail.com>
References: <a7c5abda06574a9381b815c6d80d39a5@LUCKMAN.anl.gov>
	<CAGCphBvBUd-Y0e9giMTka1QMkHeyozf21bF3yNGjc05bUiVgSg@mail.gmail.com>
Message-ID: <52DBAAC7.2010905@mcs.anl.gov>

Chapeau! :-)


On 01/19/2014 05:22 AM, Hong Zhang wrote:
> Jed,
> Congratulates!!!
> Hong
>
> ---------- Forwarded message ----------
> From: *Snir, Marc* <snir at anl.gov <mailto:snir at anl.gov>>
> Date: Sat, Jan 18, 2014 at 3:12 PM
> Subject: [MCS Staff] Jed Brown wins the SIAG/Supercomputing Junior
> Scientist Prize
> To: "mcs at mcs.anl.gov <mailto:mcs at mcs.anl.gov>" <mcs at mcs.anl.gov
> <mailto:mcs at mcs.anl.gov>>
>
>
> Please join me in congratulating Jed Brown for winning the
> SIAG/Supercomputing Junior Scientist Prize
>
> The SIAM Activity Group on Supercomputing (SIAG/SC) Junior Scientist
> Prize, established in 2009, is awarded to an outstanding junior
> researcher in the field of algorithms research and development for
> parallel scientific and engineering computing, for distinguished
> contributions to the field in the three calendar years prior to the year
> of the award. The prize will be awarded next month at the SIAM
> Conference on Parallel Processing for Scientific Computing (PP14).
>
> Thanks Jed, for your work and for the added recognition to MCS.
>
>
>
> Marc Snir
>
> Director, Mathematics and Computer Science Division,
> Argonne National Laboratory
>
> Michael Faiman and Saburo Muroga Professor,
> Department of Computer Science, University of Illinois at Urbana Champaign
>
>
>
> [Sent to announce at mcs.anl.gov <mailto:announce at mcs.anl.gov>.]
>
> _______________________________________________
> Subscription to this list is automatic based on your appointment in the
> MCS division.
>



From jed at jedbrown.org  Sun Jan 19 14:02:32 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 13:02:32 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <3BFC19AF-AF8E-4941-84DF-6CF29B9D5C31@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<3BFC19AF-AF8E-4941-84DF-6CF29B9D5C31@mcs.anl.gov>
Message-ID: <87ha8zu4dj.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    If mpicc is providing a -On flag and the PETSc ./configure user
>    provides a -Om flag should we generate a very useful error message
>    telling them of the conflict?

It is an MPI bug if the wrapper-provided flag stomps on the user flag.
(Such a bug appeared a few years ago, but was fixed then.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/855604ee/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 14:05:39 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 13:05:39 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
Message-ID: <87eh43u48c.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    Summary
>
>    The reason for the current model is that in the original plan
>    ?with-debugging=0 would providing good compiler optimization flags
>    (for each system) without the user having to set them; if the user
>    set them via COPTFLAGS then we did set them ourselves.
>
>    Observations
>
>     1) The ?standard? for passing optimization flags to ./configure is
>     CFLAGS=?-O3 crap? we don?t support this and this confuses users
>
>     2) If we followed the standard and the user only set
>     non-optimization flags with CFLAGS we would not turn on
>     optimization when user might expect it (does this ever happen?)

Does it happen and would it be confusing?  I think it is not confusing,
especially since it's what everyone else does.

>     3) We provide a different, ?better" way of providing optimization
>     flags and other CFLAGS, but know one knows about it.
>
>     4) Our ?with-debugging=0 optimization flags are pretty bad, we
>     should improve them
>
>     So choices are
>
>      A) follow standard

I'm inclined to do this because it's less code and simpler logic, thus
the easiest to explain.  If they don't set CFLAGS, of course we have to
automatically choose reasonable defaults based on --with-debugging.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/c8025b8d/attachment.sig>

From knepley at gmail.com  Sun Jan 19 14:11:10 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 14:11:10 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87eh43u48c.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
Message-ID: <CAMYG4G=bhvjb9C=vp5xi748DBLdt+X_O5S5Z-LFb74A3JR0hmA@mail.gmail.com>

On Sun, Jan 19, 2014 at 2:05 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>
> >    Summary
> >
> >    The reason for the current model is that in the original plan
> >    ?with-debugging=0 would providing good compiler optimization flags
> >    (for each system) without the user having to set them; if the user
> >    set them via COPTFLAGS then we did set them ourselves.
> >
> >    Observations
> >
> >     1) The ?standard? for passing optimization flags to ./configure is
> >     CFLAGS=?-O3 crap? we don?t support this and this confuses users
> >
> >     2) If we followed the standard and the user only set
> >     non-optimization flags with CFLAGS we would not turn on
> >     optimization when user might expect it (does this ever happen?)
>
> Does it happen and would it be confusing?  I think it is not confusing,
> especially since it's what everyone else does.
>
> >     3) We provide a different, ?better" way of providing optimization
> >     flags and other CFLAGS, but know one knows about it.
> >
> >     4) Our ?with-debugging=0 optimization flags are pretty bad, we
> >     should improve them
> >
> >     So choices are
> >
> >      A) follow standard
>
> I'm inclined to do this because it's less code and simpler logic, thus
> the easiest to explain.  If they don't set CFLAGS, of course we have to
> automatically choose reasonable defaults based on --with-debugging.
>

I think changing to this is worse than what we currently have.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/6bc3e85b/attachment.html>

From jed at jedbrown.org  Sun Jan 19 14:15:16 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 13:15:16 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <CAMYG4G=bhvjb9C=vp5xi748DBLdt+X_O5S5Z-LFb74A3JR0hmA@mail.gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<CAMYG4G=bhvjb9C=vp5xi748DBLdt+X_O5S5Z-LFb74A3JR0hmA@mail.gmail.com>
Message-ID: <878uubu3sb.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>> I'm inclined to do this because it's less code and simpler logic, thus
>> the easiest to explain.  If they don't set CFLAGS, of course we have to
>> automatically choose reasonable defaults based on --with-debugging.
>>
>
> I think changing to this is worse than what we currently have.

Why?

I just talked to Brad and he mentioned that he tried to use CFLAGS and
the optimization flags were masked.  If COPTFLAGS is not clear to an
expert like him, something is wrong with what we have.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/61d0690f/attachment.sig>

From knepley at gmail.com  Sun Jan 19 14:20:05 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 14:20:05 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <878uubu3sb.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<CAMYG4G=bhvjb9C=vp5xi748DBLdt+X_O5S5Z-LFb74A3JR0hmA@mail.gmail.com>
	<878uubu3sb.fsf@jedbrown.org>
Message-ID: <CAMYG4G=85TZJZRLuSSm3PcSTRe1BeMjkoNpFvWx8BHLLW_s4gg@mail.gmail.com>

On Sun, Jan 19, 2014 at 2:15 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> >> I'm inclined to do this because it's less code and simpler logic, thus
> >> the easiest to explain.  If they don't set CFLAGS, of course we have to
> >> automatically choose reasonable defaults based on --with-debugging.
> >>
> >
> > I think changing to this is worse than what we currently have.
>
> Why?
>
> I just talked to Brad and he mentioned that he tried to use CFLAGS and
> the optimization flags were masked.  If COPTFLAGS is not clear to an
> expert like him, something is wrong with what we have.
>

I think this is not a failure of scheme, but rather of documentation, Many
many
people also screw up the FieldSplit stuff but I still think that is right
too.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/d1431968/attachment.html>

From andrea.lani at gmail.com  Sun Jan 19 15:21:48 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Sun, 19 Jan 2014 22:21:48 +0100
Subject: [petsc-dev] petsc-dev fails test with debugging=0
Message-ID: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>

Dear Devs,

first of all, even after the latest Karl's fix, my GPU-enabled solver
(using PETSc's KSPGMRES and PCASM) still gives a substantially different
residual after one time step in 1 or more GPUs:

+0.72 (1 GPU) against -1.00 (2-8 GPU).

The situtation is even more dramatic for PCBJACOBI: in this case it works
the same on 1 GPU and crashes on more GPUs ( "Null argument, when expecting
valid pointer! [0]PETSC ERROR: Trying to zero at a null pointer!").

The only thing that could be wrong on my side, I think, is the way I
construct the matrix sparsity pattern for the MPIAIJ matrix, as far as I
see, since that is the only difference between a "normal"scenario using
parallel BAIJ matrices (which perfectly works for me on multiple CPUs).

BUT, surprise, if I use PCJACOBI (which is GPU-based as far as I
understood), single and multi-GPU residuals are identical! so I start
doubting that the problem is due to my sparsity pattern. Perhaps there is
something wrong in the PCASM / PCBJACOBI preconditioners when used combined
with the GPU-version of the KSP ... what do you think?

I remind that I create the matrix for the multi-GPU case with:

MatSetType(m_mat, MATMPIAIJCUSP);
MatSetBlockSize(m_mat, blockSize);
MatMPIAIJSetPreallocation(m_mat, dnz, dnnz, onz, onnz);

please confirm if this is right ...

I do have another problem though: if I compile the last petsc-dev master
with  debugging=0 (which is desirable in my case since I'd like to do
benchmarks), the petsc tests fail with the messages in the enclosed text
file. Any idea why this happens. Mind that I'm using openmpi version 1.6.5
(if this matters) and that by just configuring/compiling with debugging
flag = 1, all tests passed with no problem.
I guess this could potentially affect many users ...

Thanks in advance

Andrea









-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/d50b7521/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.err.nodebug_f77
Type: application/octet-stream
Size: 9604 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/d50b7521/attachment.obj>

From jed at jedbrown.org  Sun Jan 19 14:26:35 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 13:26:35 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <CAMYG4G=85TZJZRLuSSm3PcSTRe1BeMjkoNpFvWx8BHLLW_s4gg@mail.gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<CAMYG4G=bhvjb9C=vp5xi748DBLdt+X_O5S5Z-LFb74A3JR0hmA@mail.gmail.com>
	<878uubu3sb.fsf@jedbrown.org>
	<CAMYG4G=85TZJZRLuSSm3PcSTRe1BeMjkoNpFvWx8BHLLW_s4gg@mail.gmail.com>
Message-ID: <8761pfu39g.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> I think this is not a failure of scheme, but rather of documentation,

Nobody reads documentation, beyond perhaps the first couple lines if you
shove it in their face.  Is COPTFLAGS so important that we should use
our most valuable space to make its documentation prominent?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/ed4f6393/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 15:26:16 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 14:26:16 -0700
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
Message-ID: <8738kju0hz.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:

> After a brief delay, I'm back to needing finite element energy
> functions for optimization problems.  The original thread is
>
>     http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-December/014161.html
>
> That thread veered off into some more general discussions of the
> PetscFE API, and I don't think came to a specific conclusion as to the
> best way to add said energy functions.  In terms of API, what is
> missing is a function to integrate a function over a space, where the
> functions takes various field arguments and produces one or more
> scalars.  The quadrature rule is important: in my case there will be
> only one non-auxiliary field, and the integration function needs to
> use the same quadrature rule.

I would define a number of quantities and a reducer (MPI_SUM, MPI_MAX).

> I apologize if I failed to read through the previous discussion
> correctly, and/or the required function has already been written.  If
> not, I'm happy to mock something up, ideally with function signature
> suggestions from others first.

What is different from integrating a residual, apart from the result
being a set of scalars instead of a vector?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/265b76cc/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 15:28:15 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 14:28:15 -0700
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
Message-ID: <87zjmrslu8.fsf@jedbrown.org>

Andrea Lani <andrea.lani at gmail.com> writes:

> Dear Devs,
>
> first of all, even after the latest Karl's fix, my GPU-enabled solver
> (using PETSc's KSPGMRES and PCASM) still gives a substantially different
> residual after one time step in 1 or more GPUs:
>
> +0.72 (1 GPU) against -1.00 (2-8 GPU).
>
> The situtation is even more dramatic for PCBJACOBI: in this case it works
> the same on 1 GPU and crashes on more GPUs ( "Null argument, when expecting
> valid pointer! [0]PETSC ERROR: Trying to zero at a null pointer!").

*always* send the entire error message
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/1abf870f/attachment.sig>

From andrea.lani at gmail.com  Sun Jan 19 15:46:41 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Sun, 19 Jan 2014 22:46:41 +0100
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <87zjmrslu8.fsf@jedbrown.org>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
Message-ID: <CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>

Sure, this is the log file for the case with KSPGMRES and PCBJACOBI on 2
GPUs.

Andrea


On Sun, Jan 19, 2014 at 10:28 PM, Jed Brown <jed at jedbrown.org> wrote:

> Andrea Lani <andrea.lani at gmail.com> writes:
>
> > Dear Devs,
> >
> > first of all, even after the latest Karl's fix, my GPU-enabled solver
> > (using PETSc's KSPGMRES and PCASM) still gives a substantially different
> > residual after one time step in 1 or more GPUs:
> >
> > +0.72 (1 GPU) against -1.00 (2-8 GPU).
> >
> > The situtation is even more dramatic for PCBJACOBI: in this case it works
> > the same on 1 GPU and crashes on more GPUs ( "Null argument, when
> expecting
> > valid pointer! [0]PETSC ERROR: Trying to zero at a null pointer!").
>
> *always* send the entire error message
>



-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/9aa59a96/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.log
Type: text/x-log
Size: 35977 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/9aa59a96/attachment.bin>

From jed at jedbrown.org  Sun Jan 19 15:55:39 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 14:55:39 -0700
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
Message-ID: <87wqhvskkk.fsf@jedbrown.org>

Andrea Lani <andrea.lani at gmail.com> writes:

> Sure, this is the log file for the case with KSPGMRES and PCBJACOBI on 2
> GPUs.

Evidently Block Jacobi does not support GPUs because it uses VecPlaceArray.

What subdomain solver did you want to use?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/c15398d3/attachment.sig>

From andrea.lani at gmail.com  Sun Jan 19 16:00:30 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Sun, 19 Jan 2014 23:00:30 +0100
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <87wqhvskkk.fsf@jedbrown.org>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
	<87wqhvskkk.fsf@jedbrown.org>
Message-ID: <73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>

I want to use KSPGMRES

> 
> 
>> Sure, this is the log file for the case with KSPGMRES and PCBJACOBI on 2
>> GPUs.
> 
> Evidently Block Jacobi does not support GPUs because it uses VecPlaceArray.
> 
> What subdomain solver did you want to use?


From jed at jedbrown.org  Sun Jan 19 16:06:56 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 15:06:56 -0700
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
	<87wqhvskkk.fsf@jedbrown.org>
	<73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>
Message-ID: <87r483sk1r.fsf@jedbrown.org>

Andrea Lani <andrea.lani at gmail.com> writes:

> I want to use KSPGMRES

With what inside?  Jacobi?

Your outer solve was GMRES, which requires a linear preconditioner.
That implies fully converging the subdomain solver when using an
iterative method.  This is likely to be inefficient, in which case you'd
probably want to switch to FGMRES or GCR for the outer KSP.

I don't know how difficult it will be to make BJacobi support GPUs.
Could be easy, but someone has to write the code.  Karl or Dominic, have
you looked at this?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/8fd8f079/attachment.sig>

From bsmith at mcs.anl.gov  Sun Jan 19 16:34:49 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 16:34:49 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87eh43u48c.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
Message-ID: <234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>


  
On Jan 19, 2014, at 2:05 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>   Summary
>> 
>>   The reason for the current model is that in the original plan
>>   ?with-debugging=0 would providing good compiler optimization flags
>>   (for each system) without the user having to set them; if the user
>>   set them via COPTFLAGS then we did set them ourselves.
>> 
>>   Observations
>> 
>>    1) The ?standard? for passing optimization flags to ./configure is
>>    CFLAGS=?-O3 crap? we don?t support this and this confuses users
>> 
>>    2) If we followed the standard and the user only set
>>    non-optimization flags with CFLAGS we would not turn on
>>    optimization when user might expect it (does this ever happen?)
> 
> Does it happen and would it be confusing?  I think it is not confusing,
> especially since it's what everyone else does.

   User runs

     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=0  

   gets the same performance as 

     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=1

    Can?t understand why.

    

    I don?t understand the problem with doing a little bit better than ?what ever one else does?. The logic is trivial, just look for -O in CFLAGS and there is nothing to explain since users will get what they expect if they use CFLAGS=?-O3 -fast"

   Barry


> 
>>    3) We provide a different, ?better" way of providing optimization
>>    flags and other CFLAGS, but know one knows about it.
>> 
>>    4) Our ?with-debugging=0 optimization flags are pretty bad, we
>>    should improve them
>> 
>>    So choices are
>> 
>>     A) follow standard
> 
> I'm inclined to do this because it's less code and simpler logic, thus
> the easiest to explain.  If they don't set CFLAGS, of course we have to
> automatically choose reasonable defaults based on --with-debugging.



From bsmith at mcs.anl.gov  Sun Jan 19 16:57:36 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 16:57:36 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
Message-ID: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>


src/mat/examples/tests/ex18.c 

passes a PetscScalar to VecNorm.

When the hell are we going to have some simple checks on bad code being pushed in petsc?

   Barry



From jed at jedbrown.org  Sun Jan 19 16:58:31 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 15:58:31 -0700
Subject: [petsc-dev] PCBDDC
In-Reply-To: <85d6ef60b8fe49adb463b283880fab27@LUCKMAN.anl.gov>
References: <85d6ef60b8fe49adb463b283880fab27@LUCKMAN.anl.gov>
Message-ID: <87k3dvshns.fsf@jedbrown.org>

Stefano Zampini <stefano.zampini at gmail.com> writes:

> Hi,
>
> I think PCBDDC is now stable enough to be available without the
> configure switch --with-pcbddc. Could you please review and merge my
> opened pull requests (134, 135 and 136) and remove the configure
> switch together with the int64 restriction on the PCBDDC package?

I've been traveling continuously since you wrote this, but I'll merge
them soon and enable by default in 'next' to see what the nightlies have
to say.

> I would like to solve any issues coming from the nightly builds before
> the next release. If it is not safe to do this, I'm waiting for
> suggestions.

I see you're not on petsc-maint, but there was a recent support issue;
will Cc you on that.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/0573307d/attachment.sig>

From rupp at mcs.anl.gov  Sun Jan 19 17:20:12 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 00:20:12 +0100
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <87r483sk1r.fsf@jedbrown.org>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>	<87zjmrslu8.fsf@jedbrown.org>	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>	<87wqhvskkk.fsf@jedbrown.org>	<73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>
	<87r483sk1r.fsf@jedbrown.org>
Message-ID: <52DC5DAC.1030400@mcs.anl.gov>

Hi,

 >> I want to use KSPGMRES
>
> With what inside?  Jacobi?
>
> Your outer solve was GMRES, which requires a linear preconditioner.
> That implies fully converging the subdomain solver when using an
> iterative method.  This is likely to be inefficient, in which case you'd
> probably want to switch to FGMRES or GCR for the outer KSP.
>
> I don't know how difficult it will be to make BJacobi support GPUs.
> Could be easy, but someone has to write the code.  Karl or Dominic, have
> you looked at this?

It depends on the amount of stability one wants to put it. If Gauss 
without pivoting is sufficient, Block-Jacobi shouldn't be too hard for 
the block size of interest (9-by-9). However, I suppose we do want 
robust solves, so it's better to use some explicit solution formulas. 
Cramer's rule is okay up to 4-by-4 blocks or maybe 5-by-5, but likely to 
be a mess for 9-by-9.

Best regards,
Karli




From jed at jedbrown.org  Sun Jan 19 17:26:51 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 16:26:51 -0700
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <52DC5DAC.1030400@mcs.anl.gov>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
	<87wqhvskkk.fsf@jedbrown.org>
	<73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>
	<87r483sk1r.fsf@jedbrown.org> <52DC5DAC.1030400@mcs.anl.gov>
Message-ID: <87zjmrr1s4.fsf@jedbrown.org>

Karl Rupp <rupp at mcs.anl.gov> writes:
> It depends on the amount of stability one wants to put it. If Gauss 
> without pivoting is sufficient, Block-Jacobi shouldn't be too hard for 
> the block size of interest (9-by-9). 

This is called pbjacobi.  I think that is a better choice to put on the
GPU.

> However, I suppose we do want robust solves, so it's better to use
> some explicit solution formulas.  Cramer's rule is okay up to 4-by-4
> blocks or maybe 5-by-5, but likely to be a mess for 9-by-9.

PBJacobi inverts all the diagonals.  I suggest just doing that on the
CPU and then transfer the inverses to the GPU.  You'll need a custom
kernel on the GPU.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/c9600e5b/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 17:50:57 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 16:50:57 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
Message-ID: <87vbxfr0ny.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    User runs
>
>      ?with-cc=cc CFLAGS=?-m64? ?with-debugging=0
>
>    gets the same performance as
>
>      ?with-cc=cc CFLAGS=?-m64? ?with-debugging=1
>
>     Can?t understand why.

It wouldn't have quite the _same_ performance because of PETSc debugging
overhead, but point taken.  OTOH, we may be squashing optimization flags
internal to the wrappers and some compilers have funny names for
optimization flags.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/f7693959/attachment.sig>

From bsmith at mcs.anl.gov  Sun Jan 19 18:07:09 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 18:07:09 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87vbxfr0ny.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
Message-ID: <5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>


On Jan 19, 2014, at 5:50 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>   User runs
>> 
>>     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=0
>> 
>>   gets the same performance as
>> 
>>     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=1
>> 
>>    Can?t understand why.
> 
> It wouldn't have quite the _same_ performance because of PETSc debugging
> overhead, but point taken.  OTOH, we may be squashing optimization flags
> internal to the wrappers

   Nope because in my previous assignment I asked Satish to check for -O stuff in compiler wrappers as well as CFLAGS.

> and some compilers have funny names for
> optimization flags.

   This is a small possibility because normally there is a -Od as well as the funny named other stuff. If we come upon another funny name that replaces -Od we can incorporate that as well.

   Barry




From jed at jedbrown.org  Sun Jan 19 18:11:54 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 17:11:54 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
Message-ID: <87sisjqzp1.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>    Nope because in my previous assignment I asked Satish to check for
>    -O stuff in compiler wrappers as well as CFLAGS.

So if a user builds with MPICH, they'll get different optimization
options than building with Open MPI, simply because MPICH includes -O2
in the wrapper?

>> and some compilers have funny names for
>> optimization flags.
>
>    This is a small possibility because normally there is a -Od as well
>    as the funny named other stuff. If we come upon another funny name
>    that replaces -Od we can incorporate that as well.

I'm just not wild about being in the business of classifying which
options are related to optimization.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/5435e216/attachment.sig>

From knepley at gmail.com  Sun Jan 19 18:13:03 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 18:13:03 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
Message-ID: <CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>

On Sun, Jan 19, 2014 at 4:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> src/mat/examples/tests/ex18.c
>
> passes a PetscScalar to VecNorm.
>

I missed it when I went over the example, and C does not check typedefs,
only
the underlying type.


> When the hell are we going to have some simple checks on bad code being
> pushed in petsc?


I checked the compiles every day, but not the examples since there is so
much noise. So

  a) we could lump the example compiles in with the build. I like this one

  b) we can cleanup all the example noise. I have started this, but it is a
big job.

   Matt


>
>    Barry
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/d0c054ee/attachment.html>

From jed at jedbrown.org  Sun Jan 19 18:16:08 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 17:16:08 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
Message-ID: <87ppnnqzhz.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Sun, Jan 19, 2014 at 4:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> src/mat/examples/tests/ex18.c
>>
>> passes a PetscScalar to VecNorm.
>>
>
> I missed it when I went over the example, and C does not check
> typedefs, only the underlying type.

You have to do a build with complex.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/f96d3708/attachment.sig>

From knepley at gmail.com  Sun Jan 19 18:17:34 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 18:17:34 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <87ppnnqzhz.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
Message-ID: <CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>

On Sun, Jan 19, 2014 at 6:16 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Sun, Jan 19, 2014 at 4:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >>
> >> src/mat/examples/tests/ex18.c
> >>
> >> passes a PetscScalar to VecNorm.
> >>
> >
> > I missed it when I went over the example, and C does not check
> > typedefs, only the underlying type.
>
> You have to do a build with complex.
>

Yes, I am aware, and as I pointed out, I was checking the complex build,
but not the examples
because it is so noisy.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/5e41260c/attachment.html>

From balay at mcs.anl.gov  Sun Jan 19 18:23:43 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Sun, 19 Jan 2014 18:23:43 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87sisjqzp1.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401191822480.7308@asterix>

On Sun, 19 Jan 2014, Jed Brown wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
> >    Nope because in my previous assignment I asked Satish to check for
> >    -O stuff in compiler wrappers as well as CFLAGS.
> 
> So if a user builds with MPICH, they'll get different optimization
> options than building with Open MPI, simply because MPICH includes -O2
> in the wrapper?
> 
> >> and some compilers have funny names for
> >> optimization flags.
> >
> >    This is a small possibility because normally there is a -Od as well
> >    as the funny named other stuff. If we come upon another funny name
> >    that replaces -Od we can incorporate that as well.
> 
> I'm just not wild about being in the business of classifying which
> options are related to optimization.

Perhaps swapping the order from '$CFLAGS $COPTFLAGS' to '$COPTFLAGS
$CFLAGS' will suffice.

Satish


From irving at naml.us  Sun Jan 19 18:28:31 2014
From: irving at naml.us (Geoffrey Irving)
Date: Sun, 19 Jan 2014 16:28:31 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <8738kju0hz.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
Message-ID: <CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>

On Sun, Jan 19, 2014 at 1:26 PM, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>
>> After a brief delay, I'm back to needing finite element energy
>> functions for optimization problems.  The original thread is
>>
>>     http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-December/014161.html
>>
>> That thread veered off into some more general discussions of the
>> PetscFE API, and I don't think came to a specific conclusion as to the
>> best way to add said energy functions.  In terms of API, what is
>> missing is a function to integrate a function over a space, where the
>> functions takes various field arguments and produces one or more
>> scalars.  The quadrature rule is important: in my case there will be
>> only one non-auxiliary field, and the integration function needs to
>> use the same quadrature rule.
>
> I would define a number of quantities and a reducer (MPI_SUM, MPI_MAX).

For anything other than MPI_SUM, the evaluation points would likely
want to be different, so the structure of the code would be
significantly changed.  For example, MPI_MAX probably wants evaluation
points at the vertices, which can't be optimally coded using the same
code structure used for MPI_SUM (since different elements share
vertices).

>> I apologize if I failed to read through the previous discussion
>> correctly, and/or the required function has already been written.  If
>> not, I'm happy to mock something up, ideally with function signature
>> suggestions from others first.
>
> What is different from integrating a residual, apart from the result
> being a set of scalars instead of a vector?

The main difference is that the quadrature rule has to be specified
independent of the PetscFE objects, since the PetscFE elements for the
different fields might have incompatible quadrature rules.

Geoffrey


From sean.michael.farley at gmail.com  Sun Jan 19 18:31:12 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Sun, 19 Jan 2014 18:31:12 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
Message-ID: <m2a9ercx4f.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

> On Jan 19, 2014, at 5:50 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>   User runs
>>> 
>>>     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=0
>>> 
>>>   gets the same performance as
>>> 
>>>     ?with-cc=cc CFLAGS=?-m64? ?with-debugging=1
>>> 
>>>    Can?t understand why.
>> 
>> It wouldn't have quite the _same_ performance because of PETSc debugging
>> overhead, but point taken.  OTOH, we may be squashing optimization flags
>> internal to the wrappers
>
>    Nope because in my previous assignment I asked Satish to check for -O stuff in compiler wrappers as well as CFLAGS.
>
>> and some compilers have funny names for
>> optimization flags.
>
>    This is a small possibility because normally there is a -Od as well as the funny named other stuff. If we come upon another funny name that replaces -Od we can incorporate that as well.

I haven't chimed in for this thread but I'd like to point out that this
CFLAGS trickery is also a problem for package maintainers. These people
are usually admins and have no specific knowledge of PETSc
internals. Every single time a package does something outside the norm
for configure makes us curse. A lot.

For example, PETSc is the only project in all of 17963 ports of MacPorts
that doesn't accept CC, CXX, FC, etc. for changing its compiler for
configure. These types of variables are set outside of the port file
(i.e. in the base system of MacPorts) and make it annoying to deal with
PETSc.

Same goes for package layout. There are about 100 packages in MacPorts
that violate the standard layout of /include, /bin, /lib, etc. (this
means having a folder named 'conf' is not allowed). That's less than
1%. Each of these packages causes extra work for the maintainer and
makes us less likely to provide this package for other users.


From balay at mcs.anl.gov  Sun Jan 19 18:35:17 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Sun, 19 Jan 2014 18:35:17 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <m2a9ercx4f.fsf@gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
Message-ID: <alpine.LFD.2.11.1401191833450.7308@asterix>

On Sun, 19 Jan 2014, Sean Farley wrote:

> I haven't chimed in for this thread but I'd like to point out that this
> CFLAGS trickery is also a problem for package maintainers. These people
> are usually admins and have no specific knowledge of PETSc
> internals. Every single time a package does something outside the norm
> for configure makes us curse. A lot.
> 
> For example, PETSc is the only project in all of 17963 ports of MacPorts
> that doesn't accept CC, CXX, FC, etc. for changing its compiler for
> configure. These types of variables are set outside of the port file
> (i.e. in the base system of MacPorts) and make it annoying to deal with
> PETSc.

To clarify - petsc configure ignores CC etc from env - but the following is accepted.

./configure CC=gcc CXX=g++ FC=gfortran

Satish

> Same goes for package layout. There are about 100 packages in MacPorts
> that violate the standard layout of /include, /bin, /lib, etc. (this
> means having a folder named 'conf' is not allowed). That's less than
> 1%. Each of these packages causes extra work for the maintainer and
> makes us less likely to provide this package for other users.
> 



From jed at jedbrown.org  Sun Jan 19 18:42:14 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 17:42:14 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
Message-ID: <87a9erqyah.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> Yes, I am aware, and as I pointed out, I was checking the complex build,

You didn't mention "complex" in this thread, just something about C
typedefs which isn't true because you get warnings like this in C.

warning: incompatible pointer types passing 'PetscScalar *' (aka '_Complex double *') to parameter of type 'PetscReal *' (aka 'double *') [-Wincompatible-pointer-types]
    ierr = VecNorm(Z,NORM_2,&dp);CHKERRQ(ierr);                /*    dp <- z'*z = e'*A'*B'*B*A'*e'     */
                            ^~~

> but not the examples because it is so noisy.

Building a single executable for all the tests would make it trivial to
check that they all compile cleanly.  Making the runs completely clean
brings back the difficulty of comparing output.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/a17bd623/attachment.sig>

From sean.michael.farley at gmail.com  Sun Jan 19 18:43:27 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Sun, 19 Jan 2014 18:43:27 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <alpine.LFD.2.11.1401191833450.7308@asterix>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
Message-ID: <m28uubcwk0.fsf@gmail.com>


balay at mcs.anl.gov writes:

> On Sun, 19 Jan 2014, Sean Farley wrote:
>
>> I haven't chimed in for this thread but I'd like to point out that this
>> CFLAGS trickery is also a problem for package maintainers. These people
>> are usually admins and have no specific knowledge of PETSc
>> internals. Every single time a package does something outside the norm
>> for configure makes us curse. A lot.
>> 
>> For example, PETSc is the only project in all of 17963 ports of MacPorts
>> that doesn't accept CC, CXX, FC, etc. for changing its compiler for
>> configure. These types of variables are set outside of the port file
>> (i.e. in the base system of MacPorts) and make it annoying to deal with
>> PETSc.
>
> To clarify - petsc configure ignores CC etc from env - but the following is accepted.

The environment is already set in base since every other package accepts
it. You can, of course, do whatever you want. Just don't expect package
maintainers to want to spend this kind of effort.


From jed at jedbrown.org  Sun Jan 19 18:44:02 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 17:44:02 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <alpine.LFD.2.11.1401191833450.7308@asterix>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
Message-ID: <878uubqy7h.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> To clarify - petsc configure ignores CC etc from env - but the following is accepted.
>
> ./configure CC=gcc CXX=g++ FC=gfortran

Yeah, though in an effort to allow package scripts to be concise and
less error-prone, they put the system-desired CC, CXX, CFLAGS, LDFLAGS,
etc. into the environment of the packaging script.  For PETSc, this
means writing

  ./configure CC="$CC" CFLAGS="$CFLAGS" CXX="$CXXFLAGS" ....

and if you forget one, the package may not be built correctly.  It
definitely makes package maintainers' lives easier to not try to be
smart.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/d69aee9c/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 18:46:31 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 17:46:31 -0700
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
Message-ID: <8738kjqy3c.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:
> For anything other than MPI_SUM, the evaluation points would likely
> want to be different, so the structure of the code would be
> significantly changed.  For example, MPI_MAX probably wants evaluation
> points at the vertices, which can't be optimally coded using the same
> code structure used for MPI_SUM (since different elements share
> vertices).

That's fair, though extrema can also occur internally to an element.

> The main difference is that the quadrature rule has to be specified
> independent of the PetscFE objects, since the PetscFE elements for the
> different fields might have incompatible quadrature rules.

How is that different from evaluating coupling terms between fields in
different spaces?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/4ca06084/attachment.sig>

From knepley at gmail.com  Sun Jan 19 18:55:41 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 18:55:41 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <87a9erqyah.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
Message-ID: <CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>

On Sun, Jan 19, 2014 at 6:42 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > Yes, I am aware, and as I pointed out, I was checking the complex build,
>
> You didn't mention "complex" in this thread, just something about C
> typedefs which isn't true because you get warnings like this in C.
>
> warning: incompatible pointer types passing 'PetscScalar *' (aka '_Complex
> double *') to parameter of type 'PetscReal *' (aka 'double *')
> [-Wincompatible-pointer-types]
>     ierr = VecNorm(Z,NORM_2,&dp);CHKERRQ(ierr);                /*    dp <-
> z'*z = e'*A'*B'*B*A'*e'     */
>                             ^


I obviously do not get this warning.

   Matt

> but not the examples because it is so noisy.
>
> Building a single executable for all the tests would make it trivial to
> check that they all compile cleanly.  Making the runs completely clean
> brings back the difficulty of comparing output.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/9a55693c/attachment.html>

From jed at jedbrown.org  Sun Jan 19 19:00:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 18:00:58 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
Message-ID: <87ppnnpiut.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Sun, Jan 19, 2014 at 6:42 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>> > Yes, I am aware, and as I pointed out, I was checking the complex build,
>>
>> You didn't mention "complex" in this thread, just something about C
>> typedefs which isn't true because you get warnings like this in C.
>>
>> warning: incompatible pointer types passing 'PetscScalar *' (aka '_Complex
>> double *') to parameter of type 'PetscReal *' (aka 'double *')
>> [-Wincompatible-pointer-types]
>>     ierr = VecNorm(Z,NORM_2,&dp);CHKERRQ(ierr);                /*    dp <-
>> z'*z = e'*A'*B'*B*A'*e'     */
>>                             ^
>
>
> I obviously do not get this warning.

What compiler?  Both gcc and clang warn.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/1dda9d28/attachment.sig>

From irving at naml.us  Sun Jan 19 19:02:29 2014
From: irving at naml.us (Geoffrey Irving)
Date: Sun, 19 Jan 2014 17:02:29 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <8738kjqy3c.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
	<8738kjqy3c.fsf@jedbrown.org>
Message-ID: <CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>

On Sun, Jan 19, 2014 at 4:46 PM, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>> For anything other than MPI_SUM, the evaluation points would likely
>> want to be different, so the structure of the code would be
>> significantly changed.  For example, MPI_MAX probably wants evaluation
>> points at the vertices, which can't be optimally coded using the same
>> code structure used for MPI_SUM (since different elements share
>> vertices).
>
> That's fair, though extrema can also occur internally to an element.
>
>> The main difference is that the quadrature rule has to be specified
>> independent of the PetscFE objects, since the PetscFE elements for the
>> different fields might have incompatible quadrature rules.
>
> How is that different from evaluating coupling terms between fields in
> different spaces?

It isn't, the only difference is that the function I'm imagining needs
an extra quadrature rule argument.  Nothing here is complicated, so if
there are no particular opinions I'll write the function and send a
patch.  Can I write it hard coded to MPI_SUM first, and we can discuss
how to generalize it after that?  I am less clear on how to best write
the generalized version.

Geoffrey


From jed at jedbrown.org  Sun Jan 19 19:03:48 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 18:03:48 -0700
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
	<8738kjqy3c.fsf@jedbrown.org>
	<CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>
Message-ID: <87mwirpiq3.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:
> It isn't, the only difference is that the function I'm imagining needs
> an extra quadrature rule argument.  Nothing here is complicated, so if
> there are no particular opinions I'll write the function and send a
> patch.  Can I write it hard coded to MPI_SUM first, and we can discuss
> how to generalize it after that?  I am less clear on how to best write
> the generalized version.

That's fine with me.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/23231958/attachment.sig>

From knepley at gmail.com  Sun Jan 19 19:06:41 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 19:06:41 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <87ppnnpiut.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
Message-ID: <CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>

On Sun, Jan 19, 2014 at 7:00 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Sun, Jan 19, 2014 at 6:42 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Matthew Knepley <knepley at gmail.com> writes:
> >> > Yes, I am aware, and as I pointed out, I was checking the complex
> build,
> >>
> >> You didn't mention "complex" in this thread, just something about C
> >> typedefs which isn't true because you get warnings like this in C.
> >>
> >> warning: incompatible pointer types passing 'PetscScalar *' (aka
> '_Complex
> >> double *') to parameter of type 'PetscReal *' (aka 'double *')
> >> [-Wincompatible-pointer-types]
> >>     ierr = VecNorm(Z,NORM_2,&dp);CHKERRQ(ierr);                /*    dp
> <-
> >> z'*z = e'*A'*B'*B*A'*e'     */
> >>                             ^
> >
> >
> > I obviously do not get this warning.
>
> What compiler?  Both gcc and clang warn.
>

I can't believe you are interested:

i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/92c74c27/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan 19 19:12:04 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 19:12:04 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <878uubqy7h.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<878uubqy7h.fsf@jedbrown.org>
Message-ID: <8F0310B7-BC30-4809-8E03-CC5C09443305@mcs.anl.gov>


  We could provide an argument to PETSc?s configure changing it to use the environmental variables that happen to be lying around. For example,

  ./configure ?use-environmental-varibles-that-may-be-randomly-set

  The reason it is not the default is too many people have random values in their environment that they don?t know about, messing up the configure.

   Now all we need is a good name for this option

Request-assigned: Satish  allow use of environmental variables to set ./configure options flag is passed to mange this


On Jan 19, 2014, at 6:44 PM, Jed Brown <jed at jedbrown.org> wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
>> To clarify - petsc configure ignores CC etc from env - but the following is accepted.
>> 
>> ./configure CC=gcc CXX=g++ FC=gfortran
> 
> Yeah, though in an effort to allow package scripts to be concise and
> less error-prone, they put the system-desired CC, CXX, CFLAGS, LDFLAGS,
> etc. into the environment of the packaging script.  For PETSc, this
> means writing
> 
>  ./configure CC="$CC" CFLAGS="$CFLAGS" CXX="$CXXFLAGS" ....
> 
> and if you forget one, the package may not be built correctly.  It
> definitely makes package maintainers' lives easier to not try to be
> smart.



From bsmith at mcs.anl.gov  Sun Jan 19 19:13:20 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 19:13:20 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <alpine.LFD.2.11.1401191822480.7308@asterix>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401191822480.7308@asterix>
Message-ID: <C070328A-C00C-4464-B613-D3D70971F91B@mcs.anl.gov>


On Jan 19, 2014, at 6:23 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Sun, 19 Jan 2014, Jed Brown wrote:
> 
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>> 
>>>   Nope because in my previous assignment I asked Satish to check for
>>>   -O stuff in compiler wrappers as well as CFLAGS.
>> 
>> So if a user builds with MPICH, they'll get different optimization
>> options than building with Open MPI, simply because MPICH includes -O2
>> in the wrapper?
>> 
>>>> and some compilers have funny names for
>>>> optimization flags.
>>> 
>>>   This is a small possibility because normally there is a -Od as well
>>>   as the funny named other stuff. If we come upon another funny name
>>>   that replaces -Od we can incorporate that as well.
>> 
>> I'm just not wild about being in the business of classifying which
>> options are related to optimization.
> 
> Perhaps swapping the order from '$CFLAGS $COPTFLAGS' to '$COPTFLAGS
> $CFLAGS' will suffice.

   No, this needs to be done right. Turn off PETSc setting optimization flags if mpicc or CFLAGS attempts to set them.

   Barry

> 
> Satish



From jed at jedbrown.org  Sun Jan 19 19:13:06 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 18:13:06 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
Message-ID: <87ha8zpial.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>> What compiler?  Both gcc and clang warn.
>>
>
> I can't believe you are interested:
>
> i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)
> Copyright (C) 2007 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This compiler warns for me:

petsc-mini:c jedbrown$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


We have to get to the bottom of this.  What do you get when you compile
this?

$ cat complex.c
#include <complex.h>

int foo(double *);
int bar(complex double x) { return foo(&x); }

I get this with the 7-year old compiler Apple is shipping.

petsc-mini:c jedbrown$ gcc -c complex.c 
complex.c: In function 'bar':
complex.c:4: warning: passing argument 1 of 'foo' from incompatible pointer type
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/82d86e12/attachment.sig>

From knepley at gmail.com  Sun Jan 19 19:15:10 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 19:15:10 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <87ha8zpial.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
Message-ID: <CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>

On Sun, Jan 19, 2014 at 7:13 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> >> What compiler?  Both gcc and clang warn.
> >>
> >
> > I can't believe you are interested:
> >
> > i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)
> > Copyright (C) 2007 Free Software Foundation, Inc.
> > This is free software; see the source for copying conditions.  There is
> NO
> > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> PURPOSE.
>
> This compiler warns for me:
>
> petsc-mini:c jedbrown$ gcc --version
> i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build
> 5658) (LLVM build 2336.11.00)
> Copyright (C) 2007 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
> We have to get to the bottom of this.  What do you get when you compile
> this?
>

What are you talking about? Of fucking course it warns when you use
complex. I never said it did not.
I said that C does not warn when I pass a PetscScalar for a PetscReal when
they are both typedef'd
to double, which it never ever does. Jesus.

   Matt


> $ cat complex.c
> #include <complex.h>
>
> int foo(double *);
> int bar(complex double x) { return foo(&x); }
>
> I get this with the 7-year old compiler Apple is shipping.
>
> petsc-mini:c jedbrown$ gcc -c complex.c
> complex.c: In function 'bar':
> complex.c:4: warning: passing argument 1 of 'foo' from incompatible
> pointer type
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/eb23808f/attachment.html>

From knepley at gmail.com  Sun Jan 19 19:16:40 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 19 Jan 2014 19:16:40 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <C070328A-C00C-4464-B613-D3D70971F91B@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401191822480.7308@asterix>
	<C070328A-C00C-4464-B613-D3D70971F91B@mcs.anl.gov>
Message-ID: <CAMYG4GkmQkRVEtBrVeGwU0WP_5PK8C6xzgnKQbC58jPLJBmSfw@mail.gmail.com>

On Sun, Jan 19, 2014 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Jan 19, 2014, at 6:23 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
> > On Sun, 19 Jan 2014, Jed Brown wrote:
> >
> >> Barry Smith <bsmith at mcs.anl.gov> writes:
> >>
> >>>   Nope because in my previous assignment I asked Satish to check for
> >>>   -O stuff in compiler wrappers as well as CFLAGS.
> >>
> >> So if a user builds with MPICH, they'll get different optimization
> >> options than building with Open MPI, simply because MPICH includes -O2
> >> in the wrapper?
> >>
> >>>> and some compilers have funny names for
> >>>> optimization flags.
> >>>
> >>>   This is a small possibility because normally there is a -Od as well
> >>>   as the funny named other stuff. If we come upon another funny name
> >>>   that replaces -Od we can incorporate that as well.
> >>
> >> I'm just not wild about being in the business of classifying which
> >> options are related to optimization.
> >
> > Perhaps swapping the order from '$CFLAGS $COPTFLAGS' to '$COPTFLAGS
> > $CFLAGS' will suffice.
>
>    No, this needs to be done right. Turn off PETSc setting optimization
> flags if mpicc or CFLAGS attempts to set them.
>

I agree with Jed that this is problematic. We will end up fixing false
positives with this, only it will not
be a user error anymore (using CFLAGS instead of COPTFLAGS as indicated in
the docs), it will be
our error, and it will waste our time looking for it.

   Matt


>    Barry
>
> >
> > Satish
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/ca0aa192/attachment.html>

From bsmith at mcs.anl.gov  Sun Jan 19 19:18:06 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 19:18:06 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87sisjqzp1.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
Message-ID: <A3F7E0AD-5873-4646-99A2-DBFFC85471B6@mcs.anl.gov>


On Jan 19, 2014, at 6:11 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>   Nope because in my previous assignment I asked Satish to check for
>>   -O stuff in compiler wrappers as well as CFLAGS.
> 
> So if a user builds with MPICH, they'll get different optimization
> options than building with Open MPI, simply because MPICH includes -O2
> in the wrapper?

   Yes.* 

    And BTW, you do know my opinion of compiler wrappers! 

   Barry

  * But note that we have not yet discussed the case where PETSc compilers the MPI implementation and what optimization flags we pass down there.

> 
>>> and some compilers have funny names for
>>> optimization flags.
>> 
>>   This is a small possibility because normally there is a -Od as well
>>   as the funny named other stuff. If we come upon another funny name
>>   that replaces -Od we can incorporate that as well.
> 
> I'm just not wild about being in the business of classifying which
> options are related to optimization.



From bsmith at mcs.anl.gov  Sun Jan 19 19:20:05 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 19:20:05 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <alpine.LFD.2.11.1401191833450.7308@asterix>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
Message-ID: <B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>


On Jan 19, 2014, at 6:35 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Sun, 19 Jan 2014, Sean Farley wrote:
> 
>> I haven't chimed in for this thread but I'd like to point out that this
>> CFLAGS trickery is also a problem for package maintainers. These people
>> are usually admins and have no specific knowledge of PETSc
>> internals. Every single time a package does something outside the norm
>> for configure makes us curse. A lot.
>> 
>> For example, PETSc is the only project in all of 17963 ports of MacPorts
>> that doesn't accept CC, CXX, FC, etc. for changing its compiler for
>> configure. These types of variables are set outside of the port file
>> (i.e. in the base system of MacPorts) and make it annoying to deal with
>> PETSc.
> 
> To clarify - petsc configure ignores CC etc from env - but the following is accepted.
> 
> ./configure CC=gcc CXX=g++ FC=gfortran
> 
> Satish
> 
>> Same goes for package layout. There are about 100 packages in MacPorts
>> that violate the standard layout of /include, /bin, /lib, etc. (this
>> means having a folder named 'conf' is not allowed).

   Sean,

     Where should we put the files that end up in conf? Should we simply not install those? We could have a ./configure flag -install-skip-config-files

  Barry

>> That's less than
>> 1%. Each of these packages causes extra work for the maintainer and
>> makes us less likely to provide this package for other users.
>> 
> 



From bsmith at mcs.anl.gov  Sun Jan 19 19:24:53 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 19 Jan 2014 19:24:53 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <CAMYG4GkmQkRVEtBrVeGwU0WP_5PK8C6xzgnKQbC58jPLJBmSfw@mail.gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401191822480.7308@asterix>
	<C070328A-C00C-4464-B613-D3D70971F91B@mcs.anl.gov>
	<CAMYG4GkmQkRVEtBrVeGwU0WP_5PK8C6xzgnKQbC58jPLJBmSfw@mail.gmail.com>
Message-ID: <A6F7C343-2F4C-4346-8B6C-1B2B66AF7F2A@mcs.anl.gov>


On Jan 19, 2014, at 7:16 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Jan 19, 2014 at 7:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Jan 19, 2014, at 6:23 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Sun, 19 Jan 2014, Jed Brown wrote:
> >
> >> Barry Smith <bsmith at mcs.anl.gov> writes:
> >>
> >>>   Nope because in my previous assignment I asked Satish to check for
> >>>   -O stuff in compiler wrappers as well as CFLAGS.
> >>
> >> So if a user builds with MPICH, they'll get different optimization
> >> options than building with Open MPI, simply because MPICH includes -O2
> >> in the wrapper?
> >>
> >>>> and some compilers have funny names for
> >>>> optimization flags.
> >>>
> >>>   This is a small possibility because normally there is a -Od as well
> >>>   as the funny named other stuff. If we come upon another funny name
> >>>   that replaces -Od we can incorporate that as well.
> >>
> >> I'm just not wild about being in the business of classifying which
> >> options are related to optimization.
> >
> > Perhaps swapping the order from '$CFLAGS $COPTFLAGS' to '$COPTFLAGS
> > $CFLAGS' will suffice.
> 
>    No, this needs to be done right. Turn off PETSc setting optimization flags if mpicc or CFLAGS attempts to set them.
> 
> I agree with Jed that this is problematic. We will end up fixing false positives with this,

   What do you mean fix false positives? We look for -O, if it is in mpicc or CFLAGS we don?t set our own, how will there be false positives?

> only it will not
> be a user error anymore (using CFLAGS instead of COPTFLAGS as indicated in the docs), it will be
> our error, and it will waste our time looking for it.

   We will keep COPTFLAGS for people who want to use it and keep the documentation as is. But by handling -O in CFLAGS we will correctly handle 99% of the autoconf gear heads who automatically type CFLAGS=?-O3?   (As if they have a clue what optimization flag should be set)

   Look it is configuration, there is never a perfect answer. 

   Barry



> 
>    Matt
>  
>    Barry
> 
> >
> > Satish
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From jed at jedbrown.org  Sun Jan 19 21:17:47 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 20:17:47 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
	<CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
Message-ID: <877g9vpcis.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> What are you talking about? Of fucking course it warns when you use
> complex. I never said it did not.

I said that you have to build with complex and you said:

| I missed it when I went over the example, and C does not check typedefs,
| only the underlying type.

then:

| Yes, I am aware, and as I pointed out, I was checking the complex build,

I interpreted this as an assertion that you built the example with
complex, which is clearly not true.  (Yes, I also wish C had a strong typedef.)

This will be fixed when we have one command to compile all examples into
one executable.  Should I merge the branch that does that now instead of
holding off until we can run the test suite that way?  I think there is
value in just having the compiler check all the examples.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/fc319700/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 21:22:31 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 20:22:31 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <A3F7E0AD-5873-4646-99A2-DBFFC85471B6@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
	<A3F7E0AD-5873-4646-99A2-DBFFC85471B6@mcs.anl.gov>
Message-ID: <8761pfpcaw.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    Yes.* 

That leads to sprawling threads like this one.

http://www.open-mpi.org/community/lists/users/2008/10/6891.php

Being stupid might not decrease the incidence of weird behavior, but the
fewer "smart" components in the chain, the easier it is to debug.

>     And BTW, you do know my opinion of compiler wrappers! 

I have the same opinion.

>    Barry
>
>   * But note that we have not yet discussed the case where PETSc
>   compilers the MPI implementation and what optimization flags we pass
>   down there.

MPICHLIB_CFLAGS when building MPICH.  IMO, wrappers should never include
optimization flags.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/ae05091e/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 21:29:41 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 20:29:41 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
Message-ID: <8738kjpbyy.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>      Where should we put the files that end up in conf? Should we
>      simply not install those? 

I don't think we should install logs.  If it were up to me, I would put
the makefiles in $prefix/share/petsc/, but I would install a script
petsc-config that can be used to dump the configuration as well as
provide configuration (a la pkg-config).  I think we should recommend
that users use this petsc-config script instead of including makefiles.

What do you think?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/1f77cffe/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 22:05:12 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 21:05:12 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <8F0310B7-BC30-4809-8E03-CC5C09443305@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<878uubqy7h.fsf@jedbrown.org>
	<8F0310B7-BC30-4809-8E03-CC5C09443305@mcs.anl.gov>
Message-ID: <87wqhvnvrb.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>   We could provide an argument to PETSc?s configure changing it to use
>   the environmental variables that happen to be lying around. For
>   example,
>
>   ./configure ?use-environmental-varibles-that-may-be-randomly-set
>
>   The reason it is not the default is too many people have random
>   values in their environment that they don?t know about, messing up
>   the configure.

This is a real problem and lack of environment reproducibility is one of
the most insidious problems in HPC usability, so I'm reluctant to argue
for changing the default.  How about warning (as we do now), but inform
the user about --with-env-compilers, then add those as explicit
arguments to the reconfigure script?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/f26d1ad1/attachment.sig>

From jed at jedbrown.org  Sun Jan 19 22:35:17 2014
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 19 Jan 2014 21:35:17 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <A6F7C343-2F4C-4346-8B6C-1B2B66AF7F2A@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<87sisjqzp1.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401191822480.7308@asterix>
	<C070328A-C00C-4464-B613-D3D70971F91B@mcs.anl.gov>
	<CAMYG4GkmQkRVEtBrVeGwU0WP_5PK8C6xzgnKQbC58jPLJBmSfw@mail.gmail.com>
	<A6F7C343-2F4C-4346-8B6C-1B2B66AF7F2A@mcs.anl.gov>
Message-ID: <87txcznud6.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

> On Jan 19, 2014, at 7:16 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> I agree with Jed that this is problematic. We will end up fixing
>> false positives with this,
>
>    What do you mean fix false positives? We look for -O, if it is in
>    mpicc or CFLAGS we don?t set our own, how will there be false
>    positives?

Build --with-debugging using MPICH and get optimized everything (because
PETSc doesn't add -O0)?  Note that passing CFLAGS='-O2 -g'
--with-debugging makes sense to debug some things, so flags hidden in
wrapper compilers are semantically different from those passed
explicitly.

>> only it will not be a user error anymore (using CFLAGS instead of
>> COPTFLAGS as indicated in the docs), it will be our error, and it
>> will waste our time looking for it.
>
>    We will keep COPTFLAGS for people who want to use it and keep the
>    documentation as is. 

Is it really valuable enough to justify keeping it?  I thought we wanted
exactly one way to do things.

>    Look it is configuration, there is never a perfect answer.

Delete it all!  ;-)

https://www.varnish-cache.org/docs/2.1/phk/autocrap.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140119/bda975b6/attachment.sig>

From mc0710 at gmail.com  Mon Jan 20 01:56:20 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 01:56:20 -0600
Subject: [petsc-dev] Not able to compile with VecViennaCLGetArrayRead/Write
Message-ID: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>

Hi Everyone,

I can't seem to get petsc to play nice when I have something like

Vec X;
viennacl::vector<PetscScalar> *x;

VecViennaCLGetArrayWrite(X, &x);
//Launch some opencl kernels
VecViennaCLRestoreArrayWrite(X, &x);

I get the following compilation error:
error: ?VecSeqViennaCLGetArrayWrite? was not declared in this scope
error: ?VecSeqViennaCLRestoreArrayWrite? was not declared in this scope

I tried adding a header like "#include <petscviennacl.h>" (like
petsccusp.h) but it didn't work. Is there a fix to this or are those
functions not supposed to be called from user code?

Thanks,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/e5788afc/attachment.html>

From andrea.lani at gmail.com  Mon Jan 20 01:59:48 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Mon, 20 Jan 2014 08:59:48 +0100
Subject: [petsc-dev] petsc-dev fails test with debugging=0
In-Reply-To: <87zjmrr1s4.fsf@jedbrown.org>
References: <CALjjKKXxEB5_TeptK45kHX3VgPmqVkYyyO75qc50Xv1e6WDjyA@mail.gmail.com>
	<87zjmrslu8.fsf@jedbrown.org>
	<CALjjKKWqLNGuNSYb_d4oNqZjaBVMBZEVfppyp8PWk1nQ1RhDdQ@mail.gmail.com>
	<87wqhvskkk.fsf@jedbrown.org>
	<73D22E90-1B8C-4E75-99D4-7AF5BA7D3305@gmail.com>
	<87r483sk1r.fsf@jedbrown.org> <52DC5DAC.1030400@mcs.anl.gov>
	<87zjmrr1s4.fsf@jedbrown.org>
Message-ID: <EC85FE92-795D-4F10-87EE-163529C34F9A@gmail.com>

This sounds good!

In my original e-mail I was also pointing (1) a test failure of the master in optimized mode and (2) another (possible) problem: 

is PCASM supposed to be working, right now, in combination with GPU-based KSPGMRES on multi-GPU? For me, it doesn't. Note, the result is correct on single GPU but not for more.

Would it be possible to make PETSc abort after printing some "consistency-checking" messages from inside the code when somebody like me attempts to use "forbidden" or "not-yet supported" scenarios for solvers type+ PC type? instead of letting the code crunch numbers and eventually break loose...

Since all your solvers, preconditioners, etc. are ultimately identified by strings, 
this could be achievable I guess. 

It's a question that my code users often ask me too and I realize that for PETSC, with the massive number of choices you offer, could start make sense for saving debugging time... 
What do you think?

Andrea


On Jan 20, 2014, at 12:26 AM, Jed Brown <jed at jedbrown.org> wrote:

> Karl Rupp <rupp at mcs.anl.gov> writes:
>> It depends on the amount of stability one wants to put it. If Gauss 
>> without pivoting is sufficient, Block-Jacobi shouldn't be too hard for 
>> the block size of interest (9-by-9). 
> 
> This is called pbjacobi.  I think that is a better choice to put on the
> GPU.
> 
>> However, I suppose we do want robust solves, so it's better to use
>> some explicit solution formulas.  Cramer's rule is okay up to 4-by-4
>> blocks or maybe 5-by-5, but likely to be a mess for 9-by-9.
> 
> PBJacobi inverts all the diagonals.  I suggest just doing that on the
> CPU and then transfer the inverses to the GPU.  You'll need a custom
> kernel on the GPU.


From mc0710 at gmail.com  Mon Jan 20 02:10:58 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 02:10:58 -0600
Subject: [petsc-dev] VecViennaCL for vecs created using DM
Message-ID: <CAJzU6sBYkeYa_tcASUo0QT319cF-nqkYFhVpKDzQO1zZ=RHiKA@mail.gmail.com>

Hi Everyone,

I can't seem to find an option to use viennacl for vectors created using a
DM (for ex: -dm_vec_type viennacl, although there is an option for
matrices: -dm_mat_type seq/mpiaijviennacl). Could this be a bug?

Thanks,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/22eab2db/attachment.html>

From rupp at mcs.anl.gov  Mon Jan 20 03:07:35 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 10:07:35 +0100
Subject: [petsc-dev] Not able to compile with
	VecViennaCLGetArrayRead/Write
In-Reply-To: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>
References: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>
Message-ID: <52DCE757.90708@mcs.anl.gov>

Hi Mani,

 > I can't seem to get petsc to play nice when I have something like
>
> Vec X;
> viennacl::vector<PetscScalar> *x;
>
> VecViennaCLGetArrayWrite(X, &x);
> //Launch some opencl kernels
> VecViennaCLRestoreArrayWrite(X, &x);
>
> I get the following compilation error:
> error: ?VecSeqViennaCLGetArrayWrite? was not declared in this scope
> error: ?VecSeqViennaCLRestoreArrayWrite? was not declared in this scope
>
> I tried adding a header like "#include <petscviennacl.h>" (like
> petsccusp.h) but it didn't work. Is there a fix to this or are those
> functions not supposed to be called from user code?

Did you enable ViennaCL in the configure stage through e.g.
  --download-viennacl?
I also plan to push an update of the bindings to the latest ViennaCL 
release this week.

Best regards,
Karli



From rupp at mcs.anl.gov  Mon Jan 20 03:09:25 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 10:09:25 +0100
Subject: [petsc-dev] VecViennaCL for vecs created using DM
In-Reply-To: <CAJzU6sBYkeYa_tcASUo0QT319cF-nqkYFhVpKDzQO1zZ=RHiKA@mail.gmail.com>
References: <CAJzU6sBYkeYa_tcASUo0QT319cF-nqkYFhVpKDzQO1zZ=RHiKA@mail.gmail.com>
Message-ID: <52DCE7C5.6040307@mcs.anl.gov>

Hi Mani,

 > I can't seem to find an option to use viennacl for vectors created using
> a DM (for ex: -dm_vec_type viennacl, although there is an option for
> matrices: -dm_mat_type seq/mpiaijviennacl). Could this be a bug?

Sounds like a bug, yes. I'll double-check and push a fix.

Thanks and best regards,
Karli



From mc0710 at gmail.com  Mon Jan 20 03:14:00 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 03:14:00 -0600
Subject: [petsc-dev] Not able to compile with
	VecViennaCLGetArrayRead/Write
In-Reply-To: <52DCE757.90708@mcs.anl.gov>
References: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>
	<52DCE757.90708@mcs.anl.gov>
Message-ID: <CAJzU6sB5bJiqYYKH_q8rqG3Vyo38y5kzDQcfJykpQUbLJYS0vw@mail.gmail.com>

Yes, I did. The code compiles if I include
petsc/src/vec/vec/impls/seq/seqviennacl/viennaclvecimpl.h
but I'm not sure if one is supposed to do that.

Thanks,
Mani


On Mon, Jan 20, 2014 at 3:07 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > I can't seem to get petsc to play nice when I have something like
>
>>
>> Vec X;
>> viennacl::vector<PetscScalar> *x;
>>
>> VecViennaCLGetArrayWrite(X, &x);
>> //Launch some opencl kernels
>> VecViennaCLRestoreArrayWrite(X, &x);
>>
>> I get the following compilation error:
>> error: ?VecSeqViennaCLGetArrayWrite? was not declared in this scope
>> error: ?VecSeqViennaCLRestoreArrayWrite? was not declared in this scope
>>
>> I tried adding a header like "#include <petscviennacl.h>" (like
>> petsccusp.h) but it didn't work. Is there a fix to this or are those
>> functions not supposed to be called from user code?
>>
>
> Did you enable ViennaCL in the configure stage through e.g.
>  --download-viennacl?
> I also plan to push an update of the bindings to the latest ViennaCL
> release this week.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/5c152dbc/attachment.html>

From rupp at mcs.anl.gov  Mon Jan 20 03:18:28 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 10:18:28 +0100
Subject: [petsc-dev] Not able to compile with
	VecViennaCLGetArrayRead/Write
In-Reply-To: <CAJzU6sB5bJiqYYKH_q8rqG3Vyo38y5kzDQcfJykpQUbLJYS0vw@mail.gmail.com>
References: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>	<52DCE757.90708@mcs.anl.gov>
	<CAJzU6sB5bJiqYYKH_q8rqG3Vyo38y5kzDQcfJykpQUbLJYS0vw@mail.gmail.com>
Message-ID: <52DCE9E4.10500@mcs.anl.gov>

Hi,

 > Yes, I did. The code compiles if I include
> petsc/src/vec/vec/impls/seq/seqviennacl/viennaclvecimpl.h
> but I'm not sure if one is supposed to do that.

Indeed, this needs to be moved to a public include header. I'll push a 
fix shortly.

Best regards,
Karli



From rupp at mcs.anl.gov  Mon Jan 20 05:51:14 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 12:51:14 +0100
Subject: [petsc-dev] Not able to compile with
	VecViennaCLGetArrayRead/Write
In-Reply-To: <CAJzU6sB5bJiqYYKH_q8rqG3Vyo38y5kzDQcfJykpQUbLJYS0vw@mail.gmail.com>
References: <CAJzU6sC01qyGhH3p2GtfRh+aCR2d6YF0iYvuJzt1En9Q1sHHeg@mail.gmail.com>	<52DCE757.90708@mcs.anl.gov>
	<CAJzU6sB5bJiqYYKH_q8rqG3Vyo38y5kzDQcfJykpQUbLJYS0vw@mail.gmail.com>
Message-ID: <52DD0DB2.9060208@mcs.anl.gov>

Hi Mani,

VecViennaCLGetArray* is now provided through include/petscviennacl.h 
similar to how it is done for CUSP. Please use either branch 'next' or 
'karlrupp/viennacl-getarray' for this to work. Note that it uses the 
ViennaCL 1.4.2 release, I'll push updates to 1.5.0 later this week. 
Please let me know if there are any problems.

Best regards,
Karli


On 01/20/2014 10:14 AM, Mani Chandra wrote:
> Yes, I did. The code compiles if I include
> petsc/src/vec/vec/impls/seq/seqviennacl/viennaclvecimpl.h
> but I'm not sure if one is supposed to do that.
>
> Thanks,
> Mani
>
>
> On Mon, Jan 20, 2014 at 3:07 AM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Mani,
>
>
>      > I can't seem to get petsc to play nice when I have something like
>
>
>         Vec X;
>         viennacl::vector<PetscScalar> *x;
>
>         VecViennaCLGetArrayWrite(X, &x);
>         //Launch some opencl kernels
>         VecViennaCLRestoreArrayWrite(__X, &x);
>
>         I get the following compilation error:
>         error: ?VecSeqViennaCLGetArrayWrite? was not declared in this scope
>         error: ?__VecSeqViennaCLRestoreArrayWrit__e? was not declared in
>         this scope
>
>         I tried adding a header like "#include <petscviennacl.h>" (like
>         petsccusp.h) but it didn't work. Is there a fix to this or are those
>         functions not supposed to be called from user code?
>
>
>     Did you enable ViennaCL in the configure stage through e.g.
>       --download-viennacl?
>     I also plan to push an update of the bindings to the latest ViennaCL
>     release this week.
>
>     Best regards,
>     Karli
>
>



From rupp at mcs.anl.gov  Mon Jan 20 05:52:22 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 20 Jan 2014 12:52:22 +0100
Subject: [petsc-dev] VecViennaCL for vecs created using DM
In-Reply-To: <CAJzU6sBYkeYa_tcASUo0QT319cF-nqkYFhVpKDzQO1zZ=RHiKA@mail.gmail.com>
References: <CAJzU6sBYkeYa_tcASUo0QT319cF-nqkYFhVpKDzQO1zZ=RHiKA@mail.gmail.com>
Message-ID: <52DD0DF6.6030409@mcs.anl.gov>

Hi,

this is now fixed in the branches 'next' and 
'karlrupp/viennacl-getarray' and will be migrated to master in the next 
days. Note that it still uses the ViennaCL 1.4.2 release, I'll push 
updates to 1.5.0 later this week. Please let me know if there are any 
problems.

Best regards,
Karli


On 01/20/2014 09:10 AM, Mani Chandra wrote:
> Hi Everyone,
>
> I can't seem to find an option to use viennacl for vectors created using
> a DM (for ex: -dm_vec_type viennacl, although there is an option for
> matrices: -dm_mat_type seq/mpiaijviennacl). Could this be a bug?
>
> Thanks,
> Mani



From knepley at gmail.com  Mon Jan 20 07:08:59 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 20 Jan 2014 07:08:59 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <877g9vpcis.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
	<CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
	<877g9vpcis.fsf@jedbrown.org>
Message-ID: <CAMYG4GmLWrtQsv+xmE1YbfQyRu3Qq51OVDSQea1ZuVKi9=pDbw@mail.gmail.com>

On Sun, Jan 19, 2014 at 9:17 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > What are you talking about? Of fucking course it warns when you use
> > complex. I never said it did not.
>
> I said that you have to build with complex and you said:
>
> | I missed it when I went over the example, and C does not check typedefs,
> | only the underlying type.
>

No, I said the above lines first, and then you said "build it with
complex". Check the email.


> then:
>
> | Yes, I am aware, and as I pointed out, I was checking the complex build,
>
> I interpreted this as an assertion that you built the example with
> complex, which is clearly not true.  (Yes, I also wish C had a strong
> typedef.)
>

It meant I was checking the PETSc build with complex, which does not have
the examples in it.

  Matt


> This will be fixed when we have one command to compile all examples into
> one executable.  Should I merge the branch that does that now instead of
> holding off until we can run the test suite that way?  I think there is
> value in just having the compiler check all the examples.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/1ea17078/attachment.html>

From jed at jedbrown.org  Mon Jan 20 07:20:26 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 20 Jan 2014 06:20:26 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4GmLWrtQsv+xmE1YbfQyRu3Qq51OVDSQea1ZuVKi9=pDbw@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
	<CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
	<877g9vpcis.fsf@jedbrown.org>
	<CAMYG4GmLWrtQsv+xmE1YbfQyRu3Qq51OVDSQea1ZuVKi9=pDbw@mail.gmail.com>
Message-ID: <874n4yokmd.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>> This will be fixed when we have one command to compile all examples into
>> one executable.  Should I merge the branch that does that now instead of
>> holding off until we can run the test suite that way?  I think there is
>> value in just having the compiler check all the examples.

Should I do this?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/a46ea580/attachment.sig>

From knepley at gmail.com  Mon Jan 20 07:23:18 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 20 Jan 2014 07:23:18 -0600
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <874n4yokmd.fsf@jedbrown.org>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
	<CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
	<877g9vpcis.fsf@jedbrown.org>
	<CAMYG4GmLWrtQsv+xmE1YbfQyRu3Qq51OVDSQea1ZuVKi9=pDbw@mail.gmail.com>
	<874n4yokmd.fsf@jedbrown.org>
Message-ID: <CAMYG4GnU4Ds5AxPhD8C=Ft5Hp9VELMyA5g9YgarLXus+wddTpA@mail.gmail.com>

On Mon, Jan 20, 2014 at 7:20 AM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> >> This will be fixed when we have one command to compile all examples into
> >> one executable.  Should I merge the branch that does that now instead of
> >> holding off until we can run the test suite that way?  I think there is
> >> value in just having the compiler check all the examples.
>
> Should I do this?
>

definitely. I thought it worked now. I think we need a way to look at
example
compiles along with the main build.

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/f0b3abda/attachment.html>

From jedbrown at mcs.anl.gov  Mon Jan 20 07:26:46 2014
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 20 Jan 2014 06:26:46 -0700
Subject: [petsc-dev] Spectral partitioning
In-Reply-To: <3C8C8E5E-02D1-484A-AA2F-5C99BE768681@dsic.upv.es>
References: <3C8C8E5E-02D1-484A-AA2F-5C99BE768681@dsic.upv.es>
Message-ID: <871u02okbt.fsf@jedbrown.org>

"Jose E. Roman" <jroman at dsic.upv.es> writes:

> Hi Jed,
>
> I saw you added spectral partitioning
> https://bitbucket.org/petsc/petsc/commits/fc1bef75356

This was entirely based on Matt's code, which included some HSL code
that we don't have a license to include.  I refactored to avoid that.

> One thing I do not understand is why is it in MatOrdering. Shouldn't
> it be in MatPartitioning? My understanding was that MatOrdering was
> intended for fill-reducing orderings for PCFactor.
>
> I always wanted to do spectral partitioning with SLEPc. I started years ago with a Matlab script that makes spectral bisection recursively for log2(np) levels, using a generic eigensolver (eigs) in each level. It works well, but it is necessary to add a refinement step between each level (Fiduccia-Mattheyses). That is one of the reasons why I did not even start the implementation in SLEPc.
>
> On the other hand, this recent paper formulates the problem differently and uses LOBPCG to solve a generalized eigenproblem:
>
> [1] E. Vecharynski, Y. Saad, and M. Sosonkina. Graph partitioning using matrix values for preconditioning symmetric positive definite systems. SIAM Journal on Scientific Computing, 36(1):A63?A87, 2014.
> http://epubs.siam.org/doi/abs/10.1137/120898760

Interesting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/1da8c725/attachment.sig>

From jed at jedbrown.org  Mon Jan 20 07:27:53 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 20 Jan 2014 06:27:53 -0700
Subject: [petsc-dev] Matt, please don't put buggy code into PETSc
In-Reply-To: <CAMYG4GnU4Ds5AxPhD8C=Ft5Hp9VELMyA5g9YgarLXus+wddTpA@mail.gmail.com>
References: <91D1F47A-A13F-4394-BEE9-72EDC519C82B@mcs.anl.gov>
	<CAMYG4Gn9fy7qqsYxsiqFLiGHwX5SPsSoFkZT+7myLJ=yC9fTHg@mail.gmail.com>
	<87ppnnqzhz.fsf@jedbrown.org>
	<CAMYG4GnP4eVgKOdHb2cff13s99R0HOddnTfd9zwYRz00M2OVJg@mail.gmail.com>
	<87a9erqyah.fsf@jedbrown.org>
	<CAMYG4G=pu9M1zDhAb5=TqP_CuWk8r__7t6z4CwObE=HKhCAXmQ@mail.gmail.com>
	<87ppnnpiut.fsf@jedbrown.org>
	<CAMYG4GnGzboASQA0wiyUL7AW1Snvj8cG_wGaW-+Na1StyssONA@mail.gmail.com>
	<87ha8zpial.fsf@jedbrown.org>
	<CAMYG4G=+oUZErb2FEL9-qe5mGy1L2VmsMeV-qRppz=H4vOqLTQ@mail.gmail.com>
	<877g9vpcis.fsf@jedbrown.org>
	<CAMYG4GmLWrtQsv+xmE1YbfQyRu3Qq51OVDSQea1ZuVKi9=pDbw@mail.gmail.com>
	<874n4yokmd.fsf@jedbrown.org>
	<CAMYG4GnU4Ds5AxPhD8C=Ft5Hp9VELMyA5g9YgarLXus+wddTpA@mail.gmail.com>
Message-ID: <87y52an5pi.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> definitely. I thought it worked now. I think we need a way to look at
> example compiles along with the main build.

Okay, I'll do it today amid arrows and cloud bubbles.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/81e4ba6d/attachment.sig>

From knepley at gmail.com  Mon Jan 20 07:44:35 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 20 Jan 2014 07:44:35 -0600
Subject: [petsc-dev] Spectral partitioning
In-Reply-To: <871u02okbt.fsf@jedbrown.org>
References: <3C8C8E5E-02D1-484A-AA2F-5C99BE768681@dsic.upv.es>
	<871u02okbt.fsf@jedbrown.org>
Message-ID: <CAMYG4GmQmLWvk_RNFxdn-5PHS5UE6QD14h2Z51RL2_GJs_RLqw@mail.gmail.com>

On Mon, Jan 20, 2014 at 7:26 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> "Jose E. Roman" <jroman at dsic.upv.es> writes:
>
> > Hi Jed,
> >
> > I saw you added spectral partitioning
> > https://bitbucket.org/petsc/petsc/commits/fc1bef75356
>
> This was entirely based on Matt's code, which included some HSL code
> that we don't have a license to include.  I refactored to avoid that.
>

The spectral part does not use HSL code, but its entirely dense :( I would
be eager for
a SLEPc implementation, but I knew there were caveats.


> > One thing I do not understand is why is it in MatOrdering. Shouldn't
> > it be in MatPartitioning? My understanding was that MatOrdering was
> > intended for fill-reducing orderings for PCFactor.
>

So the same argument that tells you that you have a nice partition, also
tells you
that you have minimized bandwidth, so ordering makes some sense. Also, check
this out: http://arxiv.org/abs/1209.5969. I really want to do this.


> > I always wanted to do spectral partitioning with SLEPc. I started years
> ago with a Matlab script that makes spectral bisection recursively for
> log2(np) levels, using a generic eigensolver (eigs) in each level. It works
> well, but it is necessary to add a refinement step between each level
> (Fiduccia-Mattheyses). That is one of the reasons why I did not even start
> the implementation in SLEPc.
> >
> > On the other hand, this recent paper formulates the problem differently
> and uses LOBPCG to solve a generalized eigenproblem:
> >
> > [1] E. Vecharynski, Y. Saad, and M. Sosonkina. Graph partitioning using
> matrix values for preconditioning symmetric positive definite systems. SIAM
> Journal on Scientific Computing, 36(1):A63?A87, 2014.
> > http://epubs.siam.org/doi/abs/10.1137/120898760
>
> Interesting.
>

Will read.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/2e33435d/attachment.html>

From dmeiser at txcorp.com  Mon Jan 20 09:24:30 2014
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Mon, 20 Jan 2014 08:24:30 -0700
Subject: [petsc-dev] Fwd: [MCS Staff] Jed Brown wins the
 SIAG/Supercomputing Junior Scientist Prize
In-Reply-To: <52DBAAC7.2010905@mcs.anl.gov>
References: <a7c5abda06574a9381b815c6d80d39a5@LUCKMAN.anl.gov>
	<CAGCphBvBUd-Y0e9giMTka1QMkHeyozf21bF3yNGjc05bUiVgSg@mail.gmail.com>
	<52DBAAC7.2010905@mcs.anl.gov>
Message-ID: <52DD3FAE.5000002@txcorp.com>

Congrats. Well deserved.

On Sun 19 Jan 2014 03:36:55 AM MST, Karl Rupp wrote:
> Chapeau! :-)
>
>
> On 01/19/2014 05:22 AM, Hong Zhang wrote:
>> Jed,
>> Congratulates!!!
>> Hong
>>
>> ---------- Forwarded message ----------
>> From: *Snir, Marc* <snir at anl.gov <mailto:snir at anl.gov>>
>> Date: Sat, Jan 18, 2014 at 3:12 PM
>> Subject: [MCS Staff] Jed Brown wins the SIAG/Supercomputing Junior
>> Scientist Prize
>> To: "mcs at mcs.anl.gov <mailto:mcs at mcs.anl.gov>" <mcs at mcs.anl.gov
>> <mailto:mcs at mcs.anl.gov>>
>>
>>
>> Please join me in congratulating Jed Brown for winning the
>> SIAG/Supercomputing Junior Scientist Prize
>>
>> The SIAM Activity Group on Supercomputing (SIAG/SC) Junior Scientist
>> Prize, established in 2009, is awarded to an outstanding junior
>> researcher in the field of algorithms research and development for
>> parallel scientific and engineering computing, for distinguished
>> contributions to the field in the three calendar years prior to the year
>> of the award. The prize will be awarded next month at the SIAM
>> Conference on Parallel Processing for Scientific Computing (PP14).
>>
>> Thanks Jed, for your work and for the added recognition to MCS.
>>
>>
>>
>> Marc Snir
>>
>> Director, Mathematics and Computer Science Division,
>> Argonne National Laboratory
>>
>> Michael Faiman and Saburo Muroga Professor,
>> Department of Computer Science, University of Illinois at Urbana
>> Champaign
>>
>>
>>
>> [Sent to announce at mcs.anl.gov <mailto:announce at mcs.anl.gov>.]
>>
>> _______________________________________________
>> Subscription to this list is automatic based on your appointment in the
>> MCS division.
>>
>



--
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com


From wgropp at illinois.edu  Mon Jan 20 10:58:17 2014
From: wgropp at illinois.edu (William Gropp)
Date: Mon, 20 Jan 2014 08:58:17 -0800
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <871u05xk52.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<7BA562DB-1ABC-41F9-ABAC-EF61BA917B19@illinois.edu>
	<871u05xk52.fsf@jedbrown.org>
Message-ID: <9CFC1834-B23D-4B36-A697-8A9EB602EAC1@illinois.edu>

Ah, well this is the problem, and it is tough.  Users commonly use CFLAGS for three things:

1) Specify optimization levels.  This should not be included in any wrappers
2) Change the instruction set architecture (e.g., -m32 or -m64).  This must be included in the wrappers
3) Change the language accepted by the compiler (e.g.,, -std=gnu99).  This must be included in the wrappers

These really should be separate flags, but that's not going to happen.  This is why we originally introduced COPTFLAGS - so that those could be easily separated.  It is still a good idea, but it never caught on as standard practice.

Bill

William Gropp
Director, Parallel Computing Institute
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign




On Jan 18, 2014, at 9:41 AM, Jed Brown wrote:

> William Gropp <wgropp at illinois.edu> writes:
> 
>> If the MPI wrapper adds -O, its buggy.  
> 
> MPICH puts CFLAGS into the wrappers.  Since this is the standard way of
> specifying optimization when building a package, many users have
> wrappers configured this way.  MPICH must be configured using
> MPICHLIB_CFLAGS (formerly MPICH2LIB_CFLAGS) if they do not want the
> optimization to go into the wrappers.  This causes real confusion such
> as in this thread.
> 
> http://www.open-mpi.org/community/lists/users/2008/10/6924.php

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/c04acb2d/attachment.html>

From karpeev at mcs.anl.gov  Mon Jan 20 12:39:26 2014
From: karpeev at mcs.anl.gov (Dmitry Karpeyev)
Date: Mon, 20 Jan 2014 11:39:26 -0700
Subject: [petsc-dev] Jacobian checking options
In-Reply-To: <decdcad1323a4e3b9dd0ea6657c0d965@GEORGE.anl.gov>
References: <CA+Y_9U+iLu+_bnh6sowULALRhOkXiS7M9qrz3KGM1bxdK6xVaw@mail.gmail.com>
	<56ec58863ac44d5cbf036e682dd17899@NAGURSKI.anl.gov>
	<CA+Y_9U+p5VbVBtrbAh1MqRCZmtF47-_qe75iZfBx94Hm1ZF-KA@mail.gmail.com>
	<4414ca7b6152493e96e7cf2a1d6d5ea6@NAGURSKI.anl.gov>
	<CA+Y_9UK4kGjuCPM5F_B0Th3DFBkooPPe22KFC+aT6vXUEY43Qw@mail.gmail.com>
	<decdcad1323a4e3b9dd0ea6657c0d965@GEORGE.anl.gov>
Message-ID: <CA+Y_9UL8-DWgkOEvxgkNzsPiUoFuDqOFuFxQN40NBfU7qJQwUA@mail.gmail.com>

Forwarding this to petsc-dev because of (*) below.

The behavior of -snes_check_jacobian, is, unfortunately, controlled by
several options with different prefixes.
This is because a mix of PETSc objects are involved.  In particular, we
have -snes_test_xxx and -mat_fd_xxx determining the choice of the
differencing parameter h in J(u)v ~ (F(u+hv) - F(u))/h.
In particular, -mat_fd_type ds|wp controls the order of magnitude of h:
(1) wp uses h = (1+||u||)*epsilon,
(2) ds (default) uses h = u_j*epsilon, where j is the column of the
Jacobian being computed (i.e., v = e_j).

Furthermore, epsilon is controlled by -snes_test_err (and no programmatic
option?).
By default epsilon is the square root of the machine epsilon. However, if h
ends up being too small (< 1e-16, which
generally should only possible with -mat_fd_type ds), then h =
0.1*sign(u_j)*epsilon is used.

(*) Maybe control of -snes_check_jacobian, -snes_test (do we need this to
be separate from -snes_check_jacobian?  maybe this can be a suboption that
causes an early exit from the solve?), and -mat_fd can be made more
consistent?

David, are you computing the Jacobian at u = 0? Then h = 0.1*epsilon, and
the differenced residual should be h^2/h =
(0.01*machine_epsilon)/(0.1*sqrt_machine_epsilon) and should evaluate to
zero, it seems to me.

Dmitry.



On Mon, Jan 20, 2014 at 10:48 AM, Andrs, David <david.andrs at inl.gov> wrote:

>  Is there a PETSc option to control the finite differencing parameter for
> -snes_check_jacobian?
>
>  The reason why I ask is that if I have a term like: u^2 (u is the
> velocity), then the FD value for this entry is something like 1e-5, but my
> exact one is correctly zero. Then the difference from PETSc is rather
> larger then 1e-8 (which PETSc suggest is the order of the difference it
> should be).
>
>  --
> David
>
>
> On Sat, Jan 18, 2014 at 10:37 AM, Dmitry Karpeyev <karpeev at mcs.anl.gov>wrote:
>
>> You might know this already:
>> Run the smallest (fewest dofs) case possible. Identify incorrect Jacobian
>> entries. Trace them back to the faulty mesh nodes/variables. Debug the
>> corresponding Jacobian code. -start_in_debugger (-start_in_debugger lldb on
>> macos) can be helpful there.
>>
>> Let me know, if you need further help.
>> Also let me know what test cases are good for testing the element-based
>> FD Jacobian code that I am writing.
>>
>> Cheers,
>> Dmitry
>> On Jan 18, 2014 11:26 AM, "Zou (Non-US), Ling" <ling.zou at inl.gov> wrote:
>>
>>>  Dmitry, this tool is VERY helpful.
>>> I showed you last time, the first several iterations were good as I
>>> could check the Jacobians for the very first iteration. It then became
>>> worse but I don't know why since I was not able to check the Jacobians.
>>>
>>>  With this new feature, I was able to find out in later steps the
>>> hand-coded Jacobians are far away from the FD Jacobians for whatever reason
>>> (bugs I bet). I am investigating my codes.
>>>
>>>  Best,
>>>
>>>  Ling
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 14, 2014 at 5:10 PM, Dmitry Karpeyev <karpeev at mcs.anl.gov>wrote:
>>>
>>>> Sounds good.  Let me know, if you need help.
>>>>
>>>>
>>>>  On Tue, Jan 14, 2014 at 5:02 PM, Zou (Non-US), Ling <ling.zou at inl.gov>wrote:
>>>>
>>>>>  Thank you Dmitry.
>>>>> I will at first work to update my petsc version, which may need some
>>>>> help from MOOSE team though.
>>>>> I will then update you the status of the Jacobian status.
>>>>>
>>>>>  Best,
>>>>>
>>>>>  Ling
>>>>>
>>>>>
>>>>>  On Tue, Jan 14, 2014 at 4:59 PM, Dmitry Karpeyev <karpeev at mcs.anl.gov
>>>>> > wrote:
>>>>>
>>>>>> Hi Ling,
>>>>>>
>>>>>>  You can check Jacobian at EVERY SNES solve in petsc-3.4 using
>>>>>> -snes_check_jacobian
>>>>>> and view the difference between two Jacobians using
>>>>>> -snes_check_jacobian_view.
>>>>>>
>>>>>>  Naturally, this is best done on the smallest problem possible, so
>>>>>> that
>>>>>> (a) the FD Jacobian calculation doesn't take too much time,
>>>>>> (b) matrix viewing doesn't overwhelm the output.
>>>>>>
>>>>>>  MOOSE works with petsc-3.4.
>>>>>>
>>>>>>  Cheers,
>>>>>> Dmitry.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/ad2d1bcf/attachment.html>

From david.andrs at inl.gov  Mon Jan 20 12:46:19 2014
From: david.andrs at inl.gov (Andrs, David)
Date: Mon, 20 Jan 2014 11:46:19 -0700
Subject: [petsc-dev] Jacobian checking options
In-Reply-To: <CA+Y_9UL8-DWgkOEvxgkNzsPiUoFuDqOFuFxQN40NBfU7qJQwUA@mail.gmail.com>
References: <CA+Y_9U+iLu+_bnh6sowULALRhOkXiS7M9qrz3KGM1bxdK6xVaw@mail.gmail.com>
	<56ec58863ac44d5cbf036e682dd17899@NAGURSKI.anl.gov>
	<CA+Y_9U+p5VbVBtrbAh1MqRCZmtF47-_qe75iZfBx94Hm1ZF-KA@mail.gmail.com>
	<4414ca7b6152493e96e7cf2a1d6d5ea6@NAGURSKI.anl.gov>
	<CA+Y_9UK4kGjuCPM5F_B0Th3DFBkooPPe22KFC+aT6vXUEY43Qw@mail.gmail.com>
	<decdcad1323a4e3b9dd0ea6657c0d965@GEORGE.anl.gov>
	<CA+Y_9UL8-DWgkOEvxgkNzsPiUoFuDqOFuFxQN40NBfU7qJQwUA@mail.gmail.com>
Message-ID: <CADJmAN9foseZfd9M8YV6w4oUW=2PGdFL38hcJCf-UK-8qj0u+Q@mail.gmail.com>

On Mon, Jan 20, 2014 at 11:39 AM, Dmitry Karpeyev <karpeev at mcs.anl.gov>wrote:

> Forwarding this to petsc-dev because of (*) below.
>
> The behavior of -snes_check_jacobian, is, unfortunately, controlled by
> several options with different prefixes.
> This is because a mix of PETSc objects are involved.  In particular, we
> have -snes_test_xxx and -mat_fd_xxx determining the choice of the
> differencing parameter h in J(u)v ~ (F(u+hv) - F(u))/h.
> In particular, -mat_fd_type ds|wp controls the order of magnitude of h:
> (1) wp uses h = (1+||u||)*epsilon,
> (2) ds (default) uses h = u_j*epsilon, where j is the column of the
> Jacobian being computed (i.e., v = e_j).
>
> Furthermore, epsilon is controlled by -snes_test_err (and no programmatic
> option?).
> By default epsilon is the square root of the machine epsilon. However, if
> h ends up being too small (< 1e-16, which
> generally should only possible with -mat_fd_type ds), then h =
> 0.1*sign(u_j)*epsilon is used.
>
> (*) Maybe control of -snes_check_jacobian, -snes_test (do we need this to
> be separate from -snes_check_jacobian?  maybe this can be a suboption that
> causes an early exit from the solve?), and -mat_fd can be made more
> consistent?
>
> David, are you computing the Jacobian at u = 0? Then h = 0.1*epsilon, and
> the differenced residual should be h^2/h =
> (0.01*machine_epsilon)/(0.1*sqrt_machine_epsilon) and should evaluate to
> zero, it seems to me.
>
?Yes, I am.  Let me double check the code so that something else is not
kicking in...

--
David?



>
> Dmitry.
>
>
>
> On Mon, Jan 20, 2014 at 10:48 AM, Andrs, David <david.andrs at inl.gov>wrote:
>
>>  Is there a PETSc option to control the finite differencing parameter
>> for -snes_check_jacobian?
>>
>>  The reason why I ask is that if I have a term like: u^2 (u is the
>> velocity), then the FD value for this entry is something like 1e-5, but my
>> exact one is correctly zero. Then the difference from PETSc is rather
>> larger then 1e-8 (which PETSc suggest is the order of the difference it
>> should be).
>>
>>  --
>> David
>>
>>
>> On Sat, Jan 18, 2014 at 10:37 AM, Dmitry Karpeyev <karpeev at mcs.anl.gov>wrote:
>>
>>> You might know this already:
>>> Run the smallest (fewest dofs) case possible. Identify incorrect
>>> Jacobian entries. Trace them back to the faulty mesh nodes/variables. Debug
>>> the corresponding Jacobian code. -start_in_debugger (-start_in_debugger
>>> lldb on macos) can be helpful there.
>>>
>>> Let me know, if you need further help.
>>> Also let me know what test cases are good for testing the element-based
>>> FD Jacobian code that I am writing.
>>>
>>> Cheers,
>>> Dmitry
>>> On Jan 18, 2014 11:26 AM, "Zou (Non-US), Ling" <ling.zou at inl.gov> wrote:
>>>
>>>>  Dmitry, this tool is VERY helpful.
>>>> I showed you last time, the first several iterations were good as I
>>>> could check the Jacobians for the very first iteration. It then became
>>>> worse but I don't know why since I was not able to check the Jacobians.
>>>>
>>>>  With this new feature, I was able to find out in later steps the
>>>> hand-coded Jacobians are far away from the FD Jacobians for whatever reason
>>>> (bugs I bet). I am investigating my codes.
>>>>
>>>>  Best,
>>>>
>>>>  Ling
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Jan 14, 2014 at 5:10 PM, Dmitry Karpeyev <karpeev at mcs.anl.gov>wrote:
>>>>
>>>>> Sounds good.  Let me know, if you need help.
>>>>>
>>>>>
>>>>>  On Tue, Jan 14, 2014 at 5:02 PM, Zou (Non-US), Ling <ling.zou at inl.gov
>>>>> > wrote:
>>>>>
>>>>>>  Thank you Dmitry.
>>>>>> I will at first work to update my petsc version, which may need some
>>>>>> help from MOOSE team though.
>>>>>> I will then update you the status of the Jacobian status.
>>>>>>
>>>>>>  Best,
>>>>>>
>>>>>>  Ling
>>>>>>
>>>>>>
>>>>>>  On Tue, Jan 14, 2014 at 4:59 PM, Dmitry Karpeyev <
>>>>>> karpeev at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi Ling,
>>>>>>>
>>>>>>>  You can check Jacobian at EVERY SNES solve in petsc-3.4 using
>>>>>>> -snes_check_jacobian
>>>>>>> and view the difference between two Jacobians using
>>>>>>> -snes_check_jacobian_view.
>>>>>>>
>>>>>>>  Naturally, this is best done on the smallest problem possible, so
>>>>>>> that
>>>>>>> (a) the FD Jacobian calculation doesn't take too much time,
>>>>>>> (b) matrix viewing doesn't overwhelm the output.
>>>>>>>
>>>>>>>  MOOSE works with petsc-3.4.
>>>>>>>
>>>>>>>  Cheers,
>>>>>>> Dmitry.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/85c7385e/attachment.html>

From sean.michael.farley at gmail.com  Mon Jan 20 12:49:53 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 20 Jan 2014 12:49:53 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <8738kjpbyy.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org>
Message-ID: <m2zjmqbi9a.fsf@gmail.com>


jed at jedbrown.org writes:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>      Where should we put the files that end up in conf? Should we
>>      simply not install those? 
>
> I don't think we should install logs.  If it were up to me, I would put
> the makefiles in $prefix/share/petsc/, but I would install a script
> petsc-config that can be used to dump the configuration as well as
> provide configuration (a la pkg-config).  I think we should recommend
> that users use this petsc-config script instead of including makefiles.
>
> What do you think?

Jed's suggestions here would be great for package maintainers. As an
added bonus, it'd be great if $prefix/share/petsc-$foo could be be
configurable (so that multiple installations could be live
side-by-side).


From bsmith at mcs.anl.gov  Mon Jan 20 13:31:20 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 20 Jan 2014 13:31:20 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <m2zjmqbi9a.fsf@gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
Message-ID: <25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>


On Jan 20, 2014, at 12:49 PM, Sean Farley <sean.michael.farley at gmail.com> wrote:

> 
> jed at jedbrown.org writes:
> 
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>     Where should we put the files that end up in conf? Should we
>>>     simply not install those? 
>> 
>> I don't think we should install logs.  If it were up to me, I would put
>> the makefiles in $prefix/share/petsc/, but I would install a script
>> petsc-config that can be used to dump the configuration as well as
>> provide configuration (a la pkg-config).  I think we should recommend
>> that users use this petsc-config script instead of including makefiles.
>> 
>> What do you think?
> 
> Jed's suggestions here would be great for package maintainers. As an
> added bonus, it'd be great if $prefix/share/petsc-$foo could be be
> configurable (so that multiple installations could be live
> side-by-side).

    This would allow multiple copies of the ?share? stuff but if the library is in $prefix/lib/libpetsc.dylib  how can multiple installations exist side by side?

   Thanks,

     Barry



From sean.michael.farley at gmail.com  Mon Jan 20 14:27:21 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 20 Jan 2014 14:27:21 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
Message-ID: <m2y52abdqu.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

> On Jan 20, 2014, at 12:49 PM, Sean Farley <sean.michael.farley at gmail.com> wrote:
>
>> 
>> jed at jedbrown.org writes:
>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>     Where should we put the files that end up in conf? Should we
>>>>     simply not install those? 
>>> 
>>> I don't think we should install logs.  If it were up to me, I would put
>>> the makefiles in $prefix/share/petsc/, but I would install a script
>>> petsc-config that can be used to dump the configuration as well as
>>> provide configuration (a la pkg-config).  I think we should recommend
>>> that users use this petsc-config script instead of including makefiles.
>>> 
>>> What do you think?
>> 
>> Jed's suggestions here would be great for package maintainers. As an
>> added bonus, it'd be great if $prefix/share/petsc-$foo could be be
>> configurable (so that multiple installations could be live
>> side-by-side).
>
>     This would allow multiple copies of the ?share? stuff but if the library is in $prefix/lib/libpetsc.dylib  how can multiple installations exist side by side?

There would have to a configurable subprefix, e.g. include/petsc-3.4,
lib/petsc-3.4, etc. Though, it's not as big a deal to me.


From mc0710 at gmail.com  Mon Jan 20 18:44:23 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 18:44:23 -0600
Subject: [petsc-dev] No single precision support in ViennaCL?
Message-ID: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>

Hi Karl,

It looks like ViennaCL in petsc doesn't support single precision (which I
need cause my (old) GPU doesn't support double precision). I get the
following error:

Cannot use viennacl withOUT double precision numbers, it is not coded for
this capability

Is there any way this can be enabled in the build?

Thanks,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/2e8a9753/attachment.html>

From knepley at gmail.com  Mon Jan 20 19:02:48 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 20 Jan 2014 19:02:48 -0600
Subject: [petsc-dev] No single precision support in ViennaCL?
In-Reply-To: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>
References: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>
Message-ID: <CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>

On Mon, Jan 20, 2014 at 6:44 PM, Mani Chandra <mc0710 at gmail.com> wrote:

> Hi Karl,
>
> It looks like ViennaCL in petsc doesn't support single precision (which I
> need cause my (old) GPU doesn't support double precision). I get the
> following error:
>
> Cannot use viennacl withOUT double precision numbers, it is not coded for
> this capability
>
> Is there any way this can be enabled in the build?
>

This may be an oversight. Can you try adding

  self.double = 0

to the constructor in config/PETSc/packages/viennacl.py?

  Matt


> Thanks,
> Mani
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/615586a1/attachment.html>

From mc0710 at gmail.com  Mon Jan 20 19:27:34 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 19:27:34 -0600
Subject: [petsc-dev] No single precision support in ViennaCL?
In-Reply-To: <CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>
References: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>
	<CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>
Message-ID: <CAJzU6sCep8vMOJMdKd6BORxXzxsb2ABJTJrXqD9zxS3nKYcnUw@mail.gmail.com>

Thanks, that worked. I also had to add that line to
config/PETSc/packages/threadcomm.py and
config/PETSc/packages/pthreadclasses.py. I hope this won't break anything.

Cheers,
Mani


On Mon, Jan 20, 2014 at 7:02 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Jan 20, 2014 at 6:44 PM, Mani Chandra <mc0710 at gmail.com> wrote:
>
>> Hi Karl,
>>
>> It looks like ViennaCL in petsc doesn't support single precision (which I
>> need cause my (old) GPU doesn't support double precision). I get the
>> following error:
>>
>> Cannot use viennacl withOUT double precision numbers, it is not coded for
>> this capability
>>
>> Is there any way this can be enabled in the build?
>>
>
> This may be an oversight. Can you try adding
>
>   self.double = 0
>
> to the constructor in config/PETSc/packages/viennacl.py?
>
>   Matt
>
>
>> Thanks,
>> Mani
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/43b65db3/attachment.html>

From mc0710 at gmail.com  Mon Jan 20 22:47:24 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Mon, 20 Jan 2014 22:47:24 -0600
Subject: [petsc-dev] Petsc+ViennaCL usage
Message-ID: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>

Hi Everyone,

I have a few questions regarding the usage of Viennacl in Petsc.

1) In the residual evaluation function:

PetscErrorCode ComputeResidual(TS ts,
                               PetscScalar t,
                               Vec X, Vec dX_dt,
                               Vec F, void *ptr)
{
    DM da;
    Vec localX;
    TSGetDM(ts, &da)
    DMGetLocalVector(da, &localX);

    DMGlobalToLocalBegin(da, X, INSERT_VALUES, localX);
    DMGlobalToLocalEnd(da, X, INSERT_VALUES, localX);

    viennacl::vector<PetscScalar> *x, *f;
    VecViennaCLGetArrayWrite(localX, &x);
    VecViennaCLGetArrayRead(F, &f);

    viennacl::ocl::enqueue(myKernel(*x, *f));
//Should it be viennacl::ocl::enqueue(myKernel(x, f))?

    VecViennaCLRestoreArrayWrite(localX, &x);
    VecViennaCLRestoreArrayRead(F, &f);
    DMRestoreLocalVector(da, &localX);
}

Will the residual evaluation occur on the GPU/accelerator depending on
where we choose the ViennaCL array computations to occur? As I understand,
if we simply use VecGetArray in the residual evaluation function, then the
residual evaluation is still done on the CPU even though the solves are
done on the GPU.

2) How does one choose on which device the ViennaCL array computations will
occur? I was looking for some flags like -viennacl cpu/gpu/accelerator but
could not find any in -help.

3) How can one pass compiler flags when building OpenCL kernels in ViennaCL?

Thanks,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/17b43cea/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 21 03:15:59 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 21 Jan 2014 10:15:59 +0100
Subject: [petsc-dev] No single precision support in ViennaCL?
In-Reply-To: <CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>
References: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>
	<CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>
Message-ID: <52DE3ACF.90904@mcs.anl.gov>

Hi,

 >     It looks like ViennaCL in petsc doesn't support single precision
>     (which I need cause my (old) GPU doesn't support double precision).
>     I get the following error:
>
>     Cannot use viennacl withOUT double precision numbers, it is not
>     coded for this capability
>
>     Is there any way this can be enabled in the build?
>
>
> This may be an oversight. Can you try adding
>
>    self.double = 0
>
> to the constructor in config/PETSc/packages/viennacl.py?

Thanks, Matt. Do we want to provide some automatic marshalling to single 
precision if not provided by the device? For 'production use' it doesn't 
really matter, because GPUs today which offer only single precision are 
generally rather low-end and won't provide any substantial benefit over 
CPUs. However, it can be handy for debugging purposes. Any opinions?

Best regards,
Karli



From rupp at mcs.anl.gov  Tue Jan 21 03:28:28 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 21 Jan 2014 10:28:28 +0100
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>
Message-ID: <52DE3DBC.80000@mcs.anl.gov>

Hi Mani,

 > I have a few questions regarding the usage of Viennacl in Petsc.
>
> 1) In the residual evaluation function:
>
> PetscErrorCode ComputeResidual(TS ts,
>                                 PetscScalar t,
>                                 Vec X, Vec dX_dt,
>                                 Vec F, void *ptr)
> {
>      DM da;
>      Vec localX;
>      TSGetDM(ts, &da)
>      DMGetLocalVector(da, &localX);
>
>      DMGlobalToLocalBegin(da, X, INSERT_VALUES, localX);
>      DMGlobalToLocalEnd(da, X, INSERT_VALUES, localX);
>
>      viennacl::vector<PetscScalar> *x, *f;
>      VecViennaCLGetArrayWrite(localX, &x);
>      VecViennaCLGetArrayRead(F, &f);
>
>      viennacl::ocl::enqueue(myKernel(*x, *f));
> //Should it be viennacl::ocl::enqueue(myKernel(x, f))?

It should be viennacl::ocl::enqueue(myKernel(*x, *f));
Usually you also want to pass the sizes to the kernel. Don't forget to 
cast the sizes to the correct types (e.g. cl_uint).


>      VecViennaCLRestoreArrayWrite(localX, &x);
>      VecViennaCLRestoreArrayRead(F, &f);
>      DMRestoreLocalVector(da, &localX);
> }
>
> Will the residual evaluation occur on the GPU/accelerator depending on
> where we choose the ViennaCL array computations to occur? As I
> understand, if we simply use VecGetArray in the residual evaluation
> function, then the residual evaluation is still done on the CPU even
> though the solves are done on the GPU.

If you use VecViennaCLGetArrayWrite(), the data will be valid on the 
GPU, so your residual evaluation should happen in the OpenCL kernel you 
provide. This is already the case in the code snippet above.


> 2) How does one choose on which device the ViennaCL array computations
> will occur? I was looking for some flags like -viennacl
> cpu/gpu/accelerator but could not find any in -help.

Use one out of
  -viennacl_device_cpu
  -viennacl_device_gpu
  -viennacl_device_accelerator


> 3) How can one pass compiler flags when building OpenCL kernels in ViennaCL?

You could do that through the ViennaCL API directly, but I'm not sure 
whether you really want to do this. Which flags do you want to set? My 
experience is that these options have little to no effect on 
performance, particularly for the memory-bandwidth-limited case. This is 
also the reason why I haven't provided a PETSc routine for this.

Best regards,
Karli



From knepley at gmail.com  Tue Jan 21 07:43:15 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 21 Jan 2014 07:43:15 -0600
Subject: [petsc-dev] No single precision support in ViennaCL?
In-Reply-To: <52DE3ACF.90904@mcs.anl.gov>
References: <CAJzU6sDjiGxYTC3Xe+K1pT4CO-pO5JCnNYXGmt99BfMFaOd7Tg@mail.gmail.com>
	<CAMYG4Gk4xnCB0c5Jc5ue23OYpimrs4HP8sH=ZQmx0S3tac20Ew@mail.gmail.com>
	<52DE3ACF.90904@mcs.anl.gov>
Message-ID: <CAMYG4Gmi8cigoMSk4y_HxQ+Cc9OEYp-wO23K3ok6pfXNgKK0Mg@mail.gmail.com>

On Tue, Jan 21, 2014 at 3:15 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi,
>
> >     It looks like ViennaCL in petsc doesn't support single precision
>
>>     (which I need cause my (old) GPU doesn't support double precision).
>>     I get the following error:
>>
>>     Cannot use viennacl withOUT double precision numbers, it is not
>>     coded for this capability
>>
>>     Is there any way this can be enabled in the build?
>>
>>
>> This may be an oversight. Can you try adding
>>
>>    self.double = 0
>>
>> to the constructor in config/PETSc/packages/viennacl.py?
>>
>
> Thanks, Matt. Do we want to provide some automatic marshalling to single
> precision if not provided by the device? For 'production use' it doesn't
> really matter, because GPUs today which offer only single precision are
> generally rather low-end and won't provide any substantial benefit over
> CPUs. However, it can be handy for debugging purposes. Any opinions?
>

I think we should since it also a way to decrease the PCI penalty.

  Matt


> Best regards,
> Karli
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/c4a09a4d/attachment.html>

From accueilproprete at gmail.com  Tue Jan 21 08:18:29 2014
From: accueilproprete at gmail.com (Mr valsin)
Date: Tue, 21 Jan 2014 15:18:29 +0100
Subject: [petsc-dev] Presentation Accueil Proprete
Message-ID: <2d6aa6f8b11b91e31625417251cabe86@smtp11.ymlpsrv.net>

--------------------------------------------------------------------------------
Cette newsletter vous a ?t? envoy?e au format graphique HTML.
Si vous lisez cette version, alors votre logiciel de messagerie pr?f?re les e-mails au format texte.
Vous pouvez lire la version originale en ligne:
http://ymlp221.net/z3jGkb
--------------------------------------------------------------------------------


Bonjour Madame, Monsieur,

Pour en rayer l'?rosion de son chiffre d'affaire et cro?tre son
activit?,
la St? Accueil Propret? a d?cid? de proposer une offre
exceptionnelle permettant d'obtenir jusqu'? 30%de r?duction sur le
prix de ses prestations  :

* Nettoyage de bureaux
* Nettoyage de commerces/restaurants
* Nettoyage industriel
* Nettoyage d'hotels/lieux d'hebergement
* Nettoyage de fin de chantier/d?m?nagement
* Nettoyage d'immeubles
* Nettoyage de parkings

Cette offre sp?ciale permet au client, en contrepartie d'une p?riode
d'engagement ferme de douze (12) mois
de b?n?ficier

* d'un prix promotionnel sur l'ensemble des prestations convenues y
compris pour leurs besoins de remise en ?tat ou d'interventions
ponctuelles
* d'une r?duction jusqu'? 30% durant les tois (3) premiers mois.

Me tenant ? votre enti?re disposition, pour toutes informations
compl?mentaires ou demandes de devis au 01.83.74.41.86. et par mail a
: accueilproprete at gmail.com
Site Internet : www. accueil-proprete.com

Cordialement

Responsable Commercial
Accueil Propret?
32 Bd de Strasbourg
75010 Paris

_____________________________
D?sinscription / Changer d'adresse e-mail: http://ymlp221.net/ugjyssmqgsgqseubgjhwggebyeem
Powered par YourMailingListProvider

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/38cbe183/attachment.html>

From bourdin at lsu.edu  Tue Jan 21 09:30:08 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Tue, 21 Jan 2014 15:30:08 +0000
Subject: [petsc-dev] DMplex reader / viewers
Message-ID: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>

Hi,

It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
   - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
     (at least not for complex meshes generated with Cubit / Trelis).
   - Reading from or writing to exodus files is not supported.
   - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only 
     wants to read boundary values, for instance.
   - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
     and may be required to exploit the output of a computations.
   - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
     save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost. 
     It is not entirely clear to me if with this layout, it would be possible to reopen a file with a 
     different processor count.
   
I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:
   - A well documented and adopted file format that most post-processors / visualization tools can use
   - Ability to read / write individual fields
   - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
     across subdomain boundaries.
   - Ability to reopen file on a different cpu count
   - Support for higher order elements

Am I missing something? If not, we can follow up with discussion on formats and implementation.

Blaise

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From g.gorman at imperial.ac.uk  Tue Jan 21 10:23:05 2014
From: g.gorman at imperial.ac.uk (Gorman, Gerard J)
Date: Tue, 21 Jan 2014 16:23:05 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
Message-ID: <EE838CA3-FBE7-4CEF-A67F-354F14C7629F@imperial.ac.uk>

Hi Blaise

This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases. 

On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:

> Hi,
> 
> It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
>   - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>     (at least not for complex meshes generated with Cubit / Trelis).
>   - Reading from or writing to exodus files is not supported.

So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.


>   - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only 
>     wants to read boundary values, for instance.

or only writing out prognostic fields for example. 

>   - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>     and may be required to exploit the output of a computations.

Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).


>   - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>     save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost. 
>     It is not entirely clear to me if with this layout, it would be possible to reopen a file with a 
>     different processor count.

I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.

> 
> I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:

Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.

>   - A well documented and adopted file format that most post-processors / visualization tools can use

ExodusII appears to be the current favoured option.

>   - Ability to read / write individual fields
>   - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
>     across subdomain boundaries.
>   - Ability to reopen file on a different cpu count

So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
??"
# ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
# part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
# sets --enable-netcdf-4 only when HDF5 is enabled.
??"
So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.

>   - Support for higher order elements
> 
> Am I missing something? If not, we can follow up with discussion on formats and implementation.
> 

Cheers
Gerard



From knepley at gmail.com  Tue Jan 21 11:01:09 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 21 Jan 2014 11:01:09 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
Message-ID: <CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>

On Tue, Jan 21, 2014 at 9:30 AM, Blaise A Bourdin <bourdin at lsu.edu> wrote:

> Hi,
>
> It looks like DMplex is steadily gaining maturity but I/O is lagging
> behind. As far as I understand, right now, PETSc can _read_ a mesh in
> exodus format, and  write binary VTS format, but many issues remain, IMHO:
>    - The exodus reader relies on a hard-coded nodeset named ?marker?.
> Generating such a nodeset is not trivial
>      (at least not for complex meshes generated with Cubit / Trelis).
>

I will fix this right away. I will put in some registration mechanism for
labels, and we can iterate.

   - Reading from or writing to exodus files is not supported.
>

Yes, I think this is the best target. It should be similar to writing HDF5
that we do for PyLith.

   Matt


>    - The VTS viewer only allows to read and write _all_ fields in a DM.
> This may be overkill if one only
>      wants to read boundary values, for instance.
>    - The VTS viewer loses all informations on exodus nodesets and cell
> sets. These may have some significance
>      and may be required to exploit the output of a computations.
>    - VTS seems to have a concept of ?blocks?. My understanding is that the
> parallel VTS viewer uses blocks to
>      save subdomains, and that continuity of piecewise linear fields
> across subdomain boundaries is lost.
>      It is not entirely clear to me if with this layout, it would be
> possible to reopen a file with a
>      different processor count.
>
> I can dedicate some resources to improving DMplex I/O. Perhaps we can
> start a discussion by listing the desired features such readers / writers
> should have. I will pitch in by listing what matters to me:
>    - A well documented and adopted file format that most post-processors /
> visualization tools can use
>    - Ability to read / write individual fields
>    - Preserve _all_ information from the exodus file (node / side / cell
> sets), do not lose continuity of fields
>      across subdomain boundaries.
>    - Ability to reopen file on a different cpu count
>    - Support for higher order elements
>
> Am I missing something? If not, we can follow up with discussion on
> formats and implementation.
>
> Blaise
>
> --
> Department of Mathematics and Center for Computation & Technology
> Louisiana State University, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
> http://www.math.lsu.edu/~bourdin
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/b843075b/attachment.html>

From mc0710 at gmail.com  Tue Jan 21 11:21:47 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 21 Jan 2014 11:21:47 -0600
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <52DE3DBC.80000@mcs.anl.gov>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>
	<52DE3DBC.80000@mcs.anl.gov>
Message-ID: <CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>

I like to pass in "-cl-nv-verbose" for compilation on nvidia cards. Also, I
pass in parameters, for ex "-D Nx=128 -D Ny=128". I'll look at the ViennaCL
api.

Thanks,
Mani


On Tue, Jan 21, 2014 at 3:28 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > I have a few questions regarding the usage of Viennacl in Petsc.
>
>>
>> 1) In the residual evaluation function:
>>
>> PetscErrorCode ComputeResidual(TS ts,
>>                                 PetscScalar t,
>>                                 Vec X, Vec dX_dt,
>>                                 Vec F, void *ptr)
>> {
>>      DM da;
>>      Vec localX;
>>      TSGetDM(ts, &da)
>>      DMGetLocalVector(da, &localX);
>>
>>      DMGlobalToLocalBegin(da, X, INSERT_VALUES, localX);
>>      DMGlobalToLocalEnd(da, X, INSERT_VALUES, localX);
>>
>>      viennacl::vector<PetscScalar> *x, *f;
>>      VecViennaCLGetArrayWrite(localX, &x);
>>      VecViennaCLGetArrayRead(F, &f);
>>
>>      viennacl::ocl::enqueue(myKernel(*x, *f));
>> //Should it be viennacl::ocl::enqueue(myKernel(x, f))?
>>
>
> It should be viennacl::ocl::enqueue(myKernel(*x, *f));
> Usually you also want to pass the sizes to the kernel. Don't forget to
> cast the sizes to the correct types (e.g. cl_uint).
>
>
>
>       VecViennaCLRestoreArrayWrite(localX, &x);
>>      VecViennaCLRestoreArrayRead(F, &f);
>>      DMRestoreLocalVector(da, &localX);
>> }
>>
>> Will the residual evaluation occur on the GPU/accelerator depending on
>> where we choose the ViennaCL array computations to occur? As I
>> understand, if we simply use VecGetArray in the residual evaluation
>> function, then the residual evaluation is still done on the CPU even
>> though the solves are done on the GPU.
>>
>
> If you use VecViennaCLGetArrayWrite(), the data will be valid on the GPU,
> so your residual evaluation should happen in the OpenCL kernel you provide.
> This is already the case in the code snippet above.
>
>
>
>  2) How does one choose on which device the ViennaCL array computations
>> will occur? I was looking for some flags like -viennacl
>> cpu/gpu/accelerator but could not find any in -help.
>>
>
> Use one out of
>  -viennacl_device_cpu
>  -viennacl_device_gpu
>  -viennacl_device_accelerator
>
>
>
>  3) How can one pass compiler flags when building OpenCL kernels in
>> ViennaCL?
>>
>
> You could do that through the ViennaCL API directly, but I'm not sure
> whether you really want to do this. Which flags do you want to set? My
> experience is that these options have little to no effect on performance,
> particularly for the memory-bandwidth-limited case. This is also the reason
> why I haven't provided a PETSc routine for this.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/bfb0d8b5/attachment.html>

From bourdin at lsu.edu  Tue Jan 21 11:56:35 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Tue, 21 Jan 2014 17:56:35 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <EE838CA3-FBE7-4CEF-A67F-354F14C7629F@imperial.ac.uk>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<EE838CA3-FBE7-4CEF-A67F-354F14C7629F@imperial.ac.uk>
Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B@lsu.edu>


On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J <g.gorman at imperial.ac.uk> wrote:

> Hi Blaise
> 
> This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases. 
> 
> On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:
> 
>> Hi,
>> 
>> It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
>>  - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>>    (at least not for complex meshes generated with Cubit / Trelis).
>>  - Reading from or writing to exodus files is not supported.
> 
> So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.
> 
> 
>>  - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only 
>>    wants to read boundary values, for instance.
> 
> or only writing out prognostic fields for example. 
> 
>>  - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>>    and may be required to exploit the output of a computations.
> 
> Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).
> 
> 
>>  - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>>    save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost. 
>>    It is not entirely clear to me if with this layout, it would be possible to reopen a file with a 
>>    different processor count.
> 
> I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.
> 
>> 
>> I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:
> 
> Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.
I have a postdoc here who will devote some of his time to this task.

> 
>>  - A well documented and adopted file format that most post-processors / visualization tools can use
> 
> ExodusII appears to be the current favoured option.
Sadly yes? But SILO may be a close second and has a more modern interface. 


> 
>>  - Ability to read / write individual fields
>>  - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
>>    across subdomain boundaries.
>>  - Ability to reopen file on a different cpu count
> 
> So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
> ??"
> # ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
> # part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
> # sets --enable-netcdf-4 only when HDF5 is enabled.
> ??"
> So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.

I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using HDF5 as their storage format.
Exodus has an parallel extension called nemesis, but I can?t figure out how how their concept of ghost point and cells works. The documentation on this point is really unclear.

Considering the dismal state of parallel FE formats and libraries, it seems to me that one needs to chose between two options:

 a. scatter back to a single I/O node and use sequential I/O using the ordering of the original (exodus) mesh. This allows reading and writing on an arbitrary number of processors, but has potential memory footprint and performance issues. How large a mesh can we reasonably expect to be able to handle this way? 
 b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an un-partitionner as a post-processor.

Unless one can come up with a better solution than a or b, I?d like to start by writing a very simple ASCII viewer demonstrating the communication involved, then modify it to use exodus, SILO or HDF5 format.

Blaise


-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From irving at naml.us  Tue Jan 21 11:58:40 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 21 Jan 2014 09:58:40 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <87mwirpiq3.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
	<8738kjqy3c.fsf@jedbrown.org>
	<CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>
	<87mwirpiq3.fsf@jedbrown.org>
Message-ID: <CAJ1ofpd-E02zyFd2rVgeOf9KaTTiRS3agaPhqqbZGM5p1xZG0Q@mail.gmail.com>

On Sun, Jan 19, 2014 at 5:03 PM, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>> It isn't, the only difference is that the function I'm imagining needs
>> an extra quadrature rule argument.  Nothing here is complicated, so if
>> there are no particular opinions I'll write the function and send a
>> patch.  Can I write it hard coded to MPI_SUM first, and we can discuss
>> how to generalize it after that?  I am less clear on how to best write
>> the generalized version.
>
> That's fine with me.

Should the result be an array of PetscScalar or PetscReal?  My
application is for objectives, so PetscReal, but PetscScalar is more
generic.

Geoffrey


From andrea.lani at gmail.com  Tue Jan 21 12:12:00 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Tue, 21 Jan 2014 19:12:00 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
Message-ID: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>

Dear devs,

sorry to bother you with this again, but can you please give me a final
word on whether KSPGMRES in combination with PCASM is currently supposed or
not to work with MATMPIAIJCUSP on multiple-GPUs as it does (that I can
clearly see) on 1 GPU? Do you have a test for that?

Again, for what I can see, residuals are inconsistent with 1 or more GPUs
and you don't get the same discrepancy with, for instance, PCJACOBI (same
exact residuals on single and multi-GPU). This makes me think (maybe
wrongly) that my part of the interfacing with PETSc is right.

Thanks in advance

Andrea




-- 
Dr. Andrea
Lani
Senior Research Engineer, PhD
Aeronautics & Aerospace dept., CFD group
Von Karman Institute for Fluid Dynamics
Chausse de Waterloo 72,
B-1640, Rhode-Saint-Genese,  Belgium
fax  : +32-2-3599600
work : +32-2-3599769
*lani at vki.ac.be <lani at vki.ac.be>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/894cb64f/attachment.html>

From irving at naml.us  Tue Jan 21 12:12:18 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 21 Jan 2014 10:12:18 -0800
Subject: [petsc-dev] questions about PetscFEIntegrateResidual_Basic
Message-ID: <CAJ1ofpeCci_SEB2h_AmwLKiZfgA0j_j=-kZSawtTzf11vC+xcA@mail.gmail.com>

PetscFEIntegrateResidual_Basic seems to have a redundant argument
list.  It takes a single PetscFE, an array of PetscFE's, and a field
index into the array.  The single PetscFE argument is ignored.  I
would imagine that either the field index or the single PetscFE should
be eliminated.

Also, PetscFEIntegrateResidual_Basic allocates the f0 array with size

    quad.numPoints * (spatial dimension of fe[0])

but accesses it as if it had size

    quad.numPoints * (number of components of fe[field])

Am I reading this right?

Geoffrey


From bsmith at mcs.anl.gov  Tue Jan 21 13:19:01 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 21 Jan 2014 13:19:01 -0600
Subject: [petsc-dev] [petsc-users] Petsc cannot be initialized on vesta
	in some --mode options
In-Reply-To: <BAY179-W58FE42DA12A57B0D0D8C15B4A40@phx.gbl>
References: <BAY179-W58FE42DA12A57B0D0D8C15B4A40@phx.gbl>
Message-ID: <67038F4A-455F-47C3-9272-595D41CF3D67@mcs.anl.gov>


  WTF

On Jan 21, 2014, at 1:14 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Hi,
> 
>    I am trying to run a PETSc program with 1024 MPI ranks on vesta.alcf.anl.gov.  The original program which was debugged and run successfully on other clusters and on vesta with a small number of ranks included many PETSc functions to use KSP solver, but they are commented off to test the PETSc initialization. Therefore, only PetscInitialize() and PetscFinalize() and some output functions are in the program. The command to run the job is:
> 
> qsub -n <number of nodes> -t 10 --mode <ranks per node> --env "F00=a:BAR=b" ./x.r 
> 
> The total number of ranks is 1024 with different combinations of <number of nodes> and <ranks per node>, such as -n 64 --mode c16 or -n 16 --mode  64.
> 
> The results showed that PetscInitialize() cannot start the petsc process with -n 64 --mode c16 since there is no output printed to stdout.  The .cobaltlog file shows the job started but just .output file didn't record any output. The .error file is like:
> 
> 2014-01-21 16:31:50.414 (INFO ) [0x40000a3bc20] 32092:ibm.runjob.AbstractOptions: using properties file /bgsys/local/etc/bg.properties
> 2014-01-21 16:31:50.416 (INFO ) [0x40000a3bc20] 32092:ibm.runjob.AbstractOptions: max open file descriptors: 65536
> 2014-01-21 16:31:50.416 (INFO ) [0x40000a3bc20] 32092:ibm.runjob.AbstractOptions: core file limit: 18446744073709551615
> 2014-01-21 16:31:50.416 (INFO ) [0x40000a3bc20] 32092:tatu.runjob.client: scheduler job id is 154599
> 2014-01-21 16:31:50.419 (INFO ) [0x400004034e0] 32092:tatu.runjob.monitor: monitor started
> 2014-01-21 16:31:50.421 (INFO ) [0x40000a3bc20] VST-00420-11731-64:32092:ibm.runjob.client.options.Parser: set local socket to runjob_mux from properties file
> 2014-01-21 16:31:53.111 (INFO ) [0x40000a3bc20] VST-00420-11731-64:729041:ibm.runjob.client.Job: job 729041 started
> 2014-01-21 16:32:03.603 (WARN ) [0x400004034e0] 32092:tatu.runjob.monitor: tracklib terminated with exit code 1
> 2014-01-21 16:41:09.554 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: received signal 15
> 2014-01-21 16:41:09.555 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: signal sent from USER
> 2014-01-21 16:41:09.555 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: sent from pid 5894
> 2014-01-21 16:41:09.555 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: could not read /proc/5894/exe
> 2014-01-21 16:41:09.555 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: Permission denied
> 2014-01-21 16:41:09.555 (WARN ) [0x40000a3bc20] VST-00420-11731-64:ibm.runjob.LogSignalInfo: sent from uid 0 (root)
> 2014-01-21 16:41:11.248 (WARN ) [0x40000a3bc20] VST-00420-11731-64:729041:ibm.runjob.client.Job: terminated by signal 9
> 2014-01-21 16:41:11.248 (WARN ) [0x40000a3bc20] VST-00420-11731-64:729041:ibm.runjob.client.Job: abnormal termination by signal 9 from rank 720
> 2014-01-21 16:41:11.248 (INFO ) [0x40000a3bc20] tatu.runjob.client: task terminated by signal 9
> 2014-01-21 16:41:11.248 (INFO ) [0x400004034e0] 32092:tatu.runjob.monitor: monitor terminating
> 2014-01-21 16:41:11.250 (INFO ) [0x40000a3bc20] tatu.runjob.client: monitor completed
> 
> 
> The petsc can start with -n 16 --mode  64 and -n 1024 --mode c1.  I also replaced PetscInitialize()  with MPI_Init() and the program can start correctly with all combinations of the options. 
> 
> What is the reason cause this strange result? Thanks.



From tautges at mcs.anl.gov  Tue Jan 21 15:07:38 2014
From: tautges at mcs.anl.gov (Tim Tautges (ANL))
Date: Tue, 21 Jan 2014 15:07:38 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
Message-ID: <52DEE19A.30306@mcs.anl.gov>



On 01/21/2014 12:00 PM, petsc-dev-request at mcs.anl.gov wrote:

> Date: Tue, 21 Jan 2014 17:56:35 +0000
> From: Blaise A Bourdin <bourdin at lsu.edu>
> To: "Gorman, Gerard J" <g.gorman at imperial.ac.uk>
> Cc: petsc-dev <petsc-dev at mcs.anl.gov>
> Subject: Re: [petsc-dev] DMplex reader / viewers
> Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu>
> Content-Type: text/plain; charset="Windows-1252"
>
>
> On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J <g.gorman at imperial.ac.uk> wrote:
>
>> Hi Blaise
>>
>> This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases.
>>
>> On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:
>>
>>> Hi,
>>>
>>> It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
>>>   - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>>>     (at least not for complex meshes generated with Cubit / Trelis).
>>>   - Reading from or writing to exodus files is not supported.
>>
>> So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.
>>
>>
>>>   - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only
>>>     wants to read boundary values, for instance.
>>
>> or only writing out prognostic fields for example.
>>
>>>   - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>>>     and may be required to exploit the output of a computations.
>>
>> Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).
>>
>>
>>>   - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>>>     save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost.
>>>     It is not entirely clear to me if with this layout, it would be possible to reopen a file with a
>>>     different processor count.
>>
>> I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.
>>
>>>
>>> I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:
>>
>> Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.
> I have a postdoc here who will devote some of his time to this task.
>
>>
>>>   - A well documented and adopted file format that most post-processors / visualization tools can use
>>
>> ExodusII appears to be the current favoured option.
> Sadly yes? But SILO may be a close second and has a more modern interface.
>
>
>>
>>>   - Ability to read / write individual fields
>>>   - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
>>>     across subdomain boundaries.
>>>   - Ability to reopen file on a different cpu count
>>
>> So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
>> ??"
>> # ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
>> # part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
>> # sets --enable-netcdf-4 only when HDF5 is enabled.
>> ??"
>> So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.
>
> I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using HDF5 as their storage format.
> Exodus has an parallel extension called nemesis, but I can?t figure out how how their concept of ghost point and cells works. The documentation on this point is really unclear.
>
> Considering the dismal state of parallel FE formats and libraries, it seems to me that one needs to chose between two options:
>
>   a. scatter back to a single I/O node and use sequential I/O using the ordering of the original (exodus) mesh. This allows reading and writing on an arbitrary number of processors, but has potential memory footprint and performance issues. How large a mesh can we reasonably expect to be able to handle this way?
>   b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an un-partitionner as a post-processor.
>
> Unless one can come up with a better solution than a or b, I?d like to start by writing a very simple ASCII viewer demonstrating the communication involved, then modify it to use exodus, SILO or HDF5 format.
>
> Blaise
>
>

Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there are 
other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial) ExodusII, and 
also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been iffy above ~16k 
procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process of developing direct 
support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.  MOAB has pretty solid 
support for things like computing sharing and ghosting between procs and exchanging/reducing field values on those 
entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the mesh/fields through MOAB or by 
translating to VTK (also supported directly from MOAB); Visit also has a plugin you can enable.  See 
http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff is on a bitbucket 
branch of petsc owned by Vijay Mahadevan.

libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is (but 
they've been working on it recently I know).

- tim


-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706



From rupp at mcs.anl.gov  Tue Jan 21 15:28:53 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 21 Jan 2014 22:28:53 +0100
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>	<52DE3DBC.80000@mcs.anl.gov>
	<CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>
Message-ID: <52DEE695.5080200@mcs.anl.gov>

Hi Mani,

 > I like to pass in "-cl-nv-verbose" for compilation on nvidia cards.

ok, in such case you also want access to the compiler output, don't you?

> Also, I pass in parameters, for ex "-D Nx=128 -D Ny=128". I'll look at
> the ViennaCL api.

Use
   viennacl::ocl::current_context().build_options("flags_here");
and make sure you have VIENNACL_WITH_OPENCL defined for your compilation 
unit. I'll extend the PETSc API suitably when updating the ViennaCL 
bindings to the latest 1.5.1 release.

Best regards,
Karli



From rupp at mcs.anl.gov  Tue Jan 21 15:38:19 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 21 Jan 2014 22:38:19 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
Message-ID: <52DEE8CB.8060202@mcs.anl.gov>

Hi Andrea,

 > sorry to bother you with this again, but can you please give me a final
> word on whether KSPGMRES in combination with PCASM is currently supposed
> or not to work with MATMPIAIJCUSP on multiple-GPUs as it does (that I
> can clearly see) on 1 GPU? Do you have a test for that?

There are certainly examples around which allow this functionality, but 
multi-GPU algorithms are not routinely tested yet.


> Again, for what I can see, residuals are inconsistent with 1 or more
> GPUs and you don't get the same discrepancy with, for instance, PCJACOBI
> (same exact residuals on single and multi-GPU). This makes me think
> (maybe wrongly) that my part of the interfacing with PETSc is right.

Do you happen to have a simple example to reproduce? Which runtime 
options do you use/set? I'll try to reproduce this tomorrow.

Best regards,
Karli



From andrea.lani at gmail.com  Tue Jan 21 16:49:53 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Tue, 21 Jan 2014 23:49:53 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <52DEE8CB.8060202@mcs.anl.gov>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
Message-ID: <5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>

Thanks a lot Karl!

Well, I don't usually use PETSc  runtime options. I have only a subset of parameters (PC types, KSP types, etc.) which are interfaced by my solver and exposed to the end-user. Other few settings (matrix types for instance) are hardcoded in some objects (which my linear system solver interface makes use of).

I can setup a testcase, but it cannot be too simple. You'll have to install my code to run it. I can setup a script for this. If you want to give it a try, I'll send you some instructions tomorrow (here is night already).

Best regards

Andrea




On Jan 21, 2014, at 10:38 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Andrea,
> 
> > sorry to bother you with this again, but can you please give me a final
>> word on whether KSPGMRES in combination with PCASM is currently supposed
>> or not to work with MATMPIAIJCUSP on multiple-GPUs as it does (that I
>> can clearly see) on 1 GPU? Do you have a test for that?
> 
> There are certainly examples around which allow this functionality, but multi-GPU algorithms are not routinely tested yet.
> 
> 
>> Again, for what I can see, residuals are inconsistent with 1 or more
>> GPUs and you don't get the same discrepancy with, for instance, PCJACOBI
>> (same exact residuals on single and multi-GPU). This makes me think
>> (maybe wrongly) that my part of the interfacing with PETSc is right.
> 
> Do you happen to have a simple example to reproduce? Which runtime options do you use/set? I'll try to reproduce this tomorrow.
> 
> Best regards,
> Karli
> 


From bourdin at lsu.edu  Tue Jan 21 17:15:22 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Tue, 21 Jan 2014 23:15:22 +0000
Subject: [petsc-dev] How do I build a shared library for exodus?
Message-ID: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>

Hi,

For some reason, I would need to generate a shared library for exodusII, but the Makefile included in the source distribution does not have a rule for this. Is there an easy way to modify the build system package in order to generate the shared library?

Blaise

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From g.gorman at imperial.ac.uk  Tue Jan 21 17:58:11 2014
From: g.gorman at imperial.ac.uk (Gorman, Gerard J)
Date: Tue, 21 Jan 2014 23:58:11 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52DEE19A.30306@mcs.anl.gov>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
	<52DEE19A.30306@mcs.anl.gov>
Message-ID: <2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>


On 21 Jan 2014, at 21:07, Tim Tautges (ANL) <tautges at mcs.anl.gov> wrote:

> 
> 
> On 01/21/2014 12:00 PM, petsc-dev-request at mcs.anl.gov wrote:
> 
>> Date: Tue, 21 Jan 2014 17:56:35 +0000
>> From: Blaise A Bourdin <bourdin at lsu.edu>
>> To: "Gorman, Gerard J" <g.gorman at imperial.ac.uk>
>> Cc: petsc-dev <petsc-dev at mcs.anl.gov>
>> Subject: Re: [petsc-dev] DMplex reader / viewers
>> Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu>
>> Content-Type: text/plain; charset="Windows-1252"
>> 
>> 
>> On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J <g.gorman at imperial.ac.uk> wrote:
>> 
>>> Hi Blaise
>>> 
>>> This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases.
>>> 
>>> On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
>>>>  - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>>>>    (at least not for complex meshes generated with Cubit / Trelis).
>>>>  - Reading from or writing to exodus files is not supported.
>>> 
>>> So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.
>>> 
>>> 
>>>>  - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only
>>>>    wants to read boundary values, for instance.
>>> 
>>> or only writing out prognostic fields for example.
>>> 
>>>>  - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>>>>    and may be required to exploit the output of a computations.
>>> 
>>> Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).
>>> 
>>> 
>>>>  - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>>>>    save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost.
>>>>    It is not entirely clear to me if with this layout, it would be possible to reopen a file with a
>>>>    different processor count.
>>> 
>>> I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.
>>> 
>>>> 
>>>> I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:
>>> 
>>> Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.
>> I have a postdoc here who will devote some of his time to this task.
>> 
>>> 
>>>>  - A well documented and adopted file format that most post-processors / visualization tools can use
>>> 
>>> ExodusII appears to be the current favoured option.
>> Sadly yes? But SILO may be a close second and has a more modern interface.
>> 
>> 
>>> 
>>>>  - Ability to read / write individual fields
>>>>  - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
>>>>    across subdomain boundaries.
>>>>  - Ability to reopen file on a different cpu count
>>> 
>>> So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
>>> ??"
>>> # ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
>>> # part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
>>> # sets --enable-netcdf-4 only when HDF5 is enabled.
>>> ??"
>>> So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.
>> 
>> I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using HDF5 as their storage format.
>> Exodus has an parallel extension called nemesis, but I can?t figure out how how their concept of ghost point and cells works. The documentation on this point is really unclear.
>> 


I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel IO front (I?m assuming those guys also run large simulations?). That said, I see this as a two stage process: first integrate with DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of readers/writers. At least this way there will be some agility and interoperability between different formats, and it will not be too disruptive to the application codes when a different formats adopted.


>> Considering the dismal state of parallel FE formats and libraries, it seems to me that one needs to chose between two options:
>> 
>>  a. scatter back to a single I/O node and use sequential I/O using the ordering of the original (exodus) mesh. This allows reading and writing on an arbitrary number of processors, but has potential memory footprint and performance issues. How large a mesh can we reasonably expect to be able to handle this way?

Personally I would stay far away from this option. Other than being a terrible serial bottleneck, it?s a major headache when you just want to run something just a little bit bigger than what happens to fit within a single node?

>>  b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an un-partitionner as a post-processor.

I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry too much about the parallel layout although some clear optimisations are possible. In the  worst case it is a three stage process of where we perform a parallel read of the connectivity, scatter/gather for continuous numbering, parallel repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.
 
>> 
>> Unless one can come up with a better solution than a or b, I?d like to start by writing a very simple ASCII viewer demonstrating the communication involved, then modify it to use exodus, SILO or HDF5 format.
>> 
>> Blaise
>> 


>> 
> 
> Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there are other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial) ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process of developing direct support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.  MOAB has pretty solid support for things like computing sharing and ghosting between procs and exchanging/reducing field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a plugin you can enable.  See http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff is on a bitbucket branch of petsc owned by Vijay Mahadevan.

Another reason this could be of great interest is that MOAB supports (according to the docs) geometric topology which could be used when adapting the mesh on a curved surface for example - another item on my which list. Is it integrated to PETSc via the plex or does this essentially replace the functionality of the plex? Why does it break down for more than 16k procs? is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM codes such as nek500 that can run on ~1m cores?

> 
> libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is (but they've been working on it recently I know).
> 


Are there any other formats that we should be considering? It?s a few years since I tried playing about with CGNS - at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks interesting as it is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?

Cheers
Gerard



From dalcinl at gmail.com  Tue Jan 21 21:22:45 2014
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Jan 2014 01:22:45 -0200
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <8738kju0hz.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
Message-ID: <CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>

On 19 January 2014 18:26, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>
>> After a brief delay, I'm back to needing finite element energy
>> functions for optimization problems.  The original thread is
>>
>>     http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-December/014161.html
>>
>> That thread veered off into some more general discussions of the
>> PetscFE API, and I don't think came to a specific conclusion as to the
>> best way to add said energy functions.  In terms of API, what is
>> missing is a function to integrate a function over a space, where the
>> functions takes various field arguments and produces one or more
>> scalars.  The quadrature rule is important: in my case there will be
>> only one non-auxiliary field, and the integration function needs to
>> use the same quadrature rule.
>
> I would define a number of quantities and a reducer (MPI_SUM, MPI_MAX).
>

I thought to do that in PetIGA, but then realized that a reducer+eval
at quadrature points+MPI_MAX sounds weird (though I can imagine some
use cases). A reduce with MPI_MAX is a lucky consequence of computing
integrals through quadrature.

>> I apologize if I failed to read through the previous discussion
>> correctly, and/or the required function has already been written.  If
>> not, I'm happy to mock something up, ideally with function signature
>> suggestions from others first.
>
> What is different from integrating a residual, apart from the result
> being a set of scalars instead of a vector?

It is pretty much the same.


-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169


From rupp at mcs.anl.gov  Wed Jan 22 03:07:19 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 10:07:19 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
Message-ID: <52DF8A47.1060402@mcs.anl.gov>

Hi,

 > Well, I don't usually use PETSc  runtime options. I have only a 
subset of parameters (PC types, KSP types, etc.) which are interfaced by 
my solver and exposed to the end-user. Other few settings (matrix types 
for instance) are hardcoded in some objects (which my linear system 
solver interface makes use of).

Okay, which parameters are these? Apparently you set PCASM and KSPGMRES 
at some point, but are there any others I need to be aware of?


> I can setup a testcase, but it cannot be too simple. You'll have to install my code to run it. I can setup a script for this. If you want to give it a try, I'll send you some instructions tomorrow (here is night already).

I think it's easier if I try to reproduce it directly with some of our 
examples first. If I fail to reproduce it, I'll ask you for a test case :-)

Best regards,
Karli


From andrea.lani at gmail.com  Wed Jan 22 03:21:55 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Wed, 22 Jan 2014 10:21:55 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <52DF8A47.1060402@mcs.anl.gov>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
	<52DF8A47.1060402@mcs.anl.gov>
Message-ID: <4B068889-D70E-44F5-A311-361A9DDB1C1F@gmail.com>

Ok, perfect!
Once I get to the office, I will prepare a short list of all the sequence of calls to PETSc and settings that I use for that case. With that in principle, you should be able to reproduce the scenario, once you get a  sparsity pattern, an actual matrix and a rhs vector.

Actually, I can also send you files with the Petsc matrix, the vector and the sparsity pattern data on, say, 2 processors so that you can try to read them in and solve the exact same system...

Andrea

On Jan 22, 2014, at 10:07 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi,
> 
> > Well, I don't usually use PETSc  runtime options. I have only a subset of parameters (PC types, KSP types, etc.) which are interfaced by my solver and exposed to the end-user. Other few settings (matrix types for instance) are hardcoded in some objects (which my linear system solver interface makes use of).
> 
> Okay, which parameters are these? Apparently you set PCASM and KSPGMRES at some point, but are there any others I need to be aware of?
> 
> 
>> I can setup a testcase, but it cannot be too simple. You'll have to install my code to run it. I can setup a script for this. If you want to give it a try, I'll send you some instructions tomorrow (here is night already).
> 
> I think it's easier if I try to reproduce it directly with some of our examples first. If I fail to reproduce it, I'll ask you for a test case :-)
> 
> Best regards,
> Karli


From rupp at mcs.anl.gov  Wed Jan 22 03:43:38 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 10:43:38 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <4B068889-D70E-44F5-A311-361A9DDB1C1F@gmail.com>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
	<52DF8A47.1060402@mcs.anl.gov>
	<4B068889-D70E-44F5-A311-361A9DDB1C1F@gmail.com>
Message-ID: <52DF92CA.9060707@mcs.anl.gov>

Hi,

 > Once I get to the office, I will prepare a short list of all the 
sequence of calls to PETSc and settings that I use for that case. With 
that in principle, you should be able to reproduce the scenario, once 
you get a  sparsity pattern, an actual matrix and a rhs vector.

Thanks. I hope that the options suffice in order to reproduce it with 
one of our examples.


> Actually, I can also send you files with the Petsc matrix, the vector and the sparsity pattern data on, say, 2 processors so that you can try to read them in and solve the exact same system...

I'll try to reproduce with our examples first and let you know if I need 
the vector and matrix data.

Best regards,
Karli


> On Jan 22, 2014, at 10:07 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:
>
>> Hi,
>>
>>> Well, I don't usually use PETSc  runtime options. I have only a subset of parameters (PC types, KSP types, etc.) which are interfaced by my solver and exposed to the end-user. Other few settings (matrix types for instance) are hardcoded in some objects (which my linear system solver interface makes use of).
>>
>> Okay, which parameters are these? Apparently you set PCASM and KSPGMRES at some point, but are there any others I need to be aware of?
>>
>>
>>> I can setup a testcase, but it cannot be too simple. You'll have to install my code to run it. I can setup a script for this. If you want to give it a try, I'll send you some instructions tomorrow (here is night already).
>>
>> I think it's easier if I try to reproduce it directly with some of our examples first. If I fail to reproduce it, I'll ask you for a test case :-)
>>
>> Best regards,
>> Karli



From g.gorman at imperial.ac.uk  Wed Jan 22 05:32:36 2014
From: g.gorman at imperial.ac.uk (Gorman, Gerard J)
Date: Wed, 22 Jan 2014 11:32:36 +0000
Subject: [petsc-dev] Fwd:  DMplex reader / viewers
References: <2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
Message-ID: <02EA64A5-1D19-46F7-8425-A0FB86F352F3@imperial.ac.uk>

Hi Garth

I was wondering what your view was on this whole discussion since you and Richard integrated XDMF with Dolfin in a dCSE last year (http://www.hector.ac.uk/cse/distributedcse/reports/UniDOLFIN/UniDOLFIN/index.html). With the benefit of hindsight are you happy with the approach you took? Was the XDMF metadata expressive enough? It seems like you must have ran into some problems as IO for vectors was not supported for example.

Cheers
Gerard


Begin forwarded message:

From: "Gorman, Gerard J" <g.gorman at imperial.ac.uk<mailto:g.gorman at imperial.ac.uk>>
Subject: Re: [petsc-dev] DMplex reader / viewers
Date: 21 January 2014 23:58:11 GMT
To: "Tim Tautges (ANL)" <tautges at mcs.anl.gov<mailto:tautges at mcs.anl.gov>>
Cc: petsc-dev <petsc-dev at mcs.anl.gov<mailto:petsc-dev at mcs.anl.gov>>


On 21 Jan 2014, at 21:07, Tim Tautges (ANL) <tautges at mcs.anl.gov<mailto:tautges at mcs.anl.gov>> wrote:



On 01/21/2014 12:00 PM, petsc-dev-request at mcs.anl.gov<mailto:petsc-dev-request at mcs.anl.gov> wrote:

Date: Tue, 21 Jan 2014 17:56:35 +0000
From: Blaise A Bourdin <bourdin at lsu.edu<mailto:bourdin at lsu.edu>>
To: "Gorman, Gerard J" <g.gorman at imperial.ac.uk<mailto:g.gorman at imperial.ac.uk>>
Cc: petsc-dev <petsc-dev at mcs.anl.gov<mailto:petsc-dev at mcs.anl.gov>>
Subject: Re: [petsc-dev] DMplex reader / viewers
Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu<mailto:ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu>>
Content-Type: text/plain; charset="Windows-1252"


On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J <g.gorman at imperial.ac.uk<mailto:g.gorman at imperial.ac.uk>> wrote:

Hi Blaise

This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases.

On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu<mailto:bourdin at lsu.edu>> wrote:

Hi,

It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
- The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
  (at least not for complex meshes generated with Cubit / Trelis).
- Reading from or writing to exodus files is not supported.

So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.


- The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only
  wants to read boundary values, for instance.

or only writing out prognostic fields for example.

- The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
  and may be required to exploit the output of a computations.

Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).


- VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
  save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost.
  It is not entirely clear to me if with this layout, it would be possible to reopen a file with a
  different processor count.

I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.


I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:

Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.
I have a postdoc here who will devote some of his time to this task.


- A well documented and adopted file format that most post-processors / visualization tools can use

ExodusII appears to be the current favoured option.
Sadly yes? But SILO may be a close second and has a more modern interface.



- Ability to read / write individual fields
- Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
  across subdomain boundaries.
- Ability to reopen file on a different cpu count

So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
??"
# ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
# part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
# sets --enable-netcdf-4 only when HDF5 is enabled.
??"
So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.

I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using HDF5 as their storage format.
Exodus has an parallel extension called nemesis, but I can?t figure out how how their concept of ghost point and cells works. The documentation on this point is really unclear.



I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel IO front (I?m assuming those guys also run large simulations?). That said, I see this as a two stage process: first integrate with DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of readers/writers. At least this way there will be some agility and interoperability between different formats, and it will not be too disruptive to the application codes when a different formats adopted.


Considering the dismal state of parallel FE formats and libraries, it seems to me that one needs to chose between two options:

a. scatter back to a single I/O node and use sequential I/O using the ordering of the original (exodus) mesh. This allows reading and writing on an arbitrary number of processors, but has potential memory footprint and performance issues. How large a mesh can we reasonably expect to be able to handle this way?

Personally I would stay far away from this option. Other than being a terrible serial bottleneck, it?s a major headache when you just want to run something just a little bit bigger than what happens to fit within a single node?

b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an un-partitionner as a post-processor.

I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry too much about the parallel layout although some clear optimisations are possible. In the  worst case it is a three stage process of where we perform a parallel read of the connectivity, scatter/gather for continuous numbering, parallel repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.


Unless one can come up with a better solution than a or b, I?d like to start by writing a very simple ASCII viewer demonstrating the communication involved, then modify it to use exodus, SILO or HDF5 format.

Blaise





Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there are other options as well. One that we're developing, based on the MOAB library, can read/write (in serial) ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process of developing direct support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.  MOAB has pretty solid support for things like computing sharing and ghosting between procs and exchanging/reducing field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a plugin you can enable.  See http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff is on a bitbucket branch of petsc owned by Vijay Mahadevan.

Another reason this could be of great interest is that MOAB supports (according to the docs) geometric topology which could be used when adapting the mesh on a curved surface for example - another item on my which list. Is it integrated to PETSc via the plex or does this essentially replace the functionality of the plex? Why does it break down for more than 16k procs? is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM codes such as nek500 that can run on ~1m cores?


libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is (but they've been working on it recently I know).



Are there any other formats that we should be considering? It?s a few years since I tried playing about with CGNS - at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks interesting as it is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?

Cheers
Gerard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/c7626934/attachment.html>

From gnw20 at cam.ac.uk  Wed Jan 22 05:56:28 2014
From: gnw20 at cam.ac.uk (Garth N. Wells)
Date: Wed, 22 Jan 2014 11:56:28 +0000
Subject: [petsc-dev] Fwd:  DMplex reader / viewers
In-Reply-To: <02EA64A5-1D19-46F7-8425-A0FB86F352F3@imperial.ac.uk>
References: <2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
	<02EA64A5-1D19-46F7-8425-A0FB86F352F3@imperial.ac.uk>
Message-ID: <36e09ff48244638c37fca8f87abebef4@cam.ac.uk>

On 2014-01-22 11:32, Gorman, Gerard J wrote:
> Hi Garth
> 
> I was wondering what your view was on this whole discussion since you
> and Richard integrated XDMF with Dolfin in a dCSE last year
> (http://www.hector.ac.uk/cse/distributedcse/reports/UniDOLFIN/UniDOLFIN/index.html
> [2]). With the benefit of hindsight are you happy with the approach
> you took? Was the XDMF metadata expressive enough?

XDMF is great for post-processing (it's supported by ParaView and 
VisIt), and it's good for reading in vanilla meshes (i.e., no markers). 
It's much, much better than the native VTK formats for visualisation - 
it's two files only and for time dependent problems the mesh only needs 
to be written once. There are some issues with how it orders points - 
the index of a vertex is implicitly taken as its position in the file. 
This isn't ideal if a user wants to inspect a vertex or cell with a 
particular index in ParaView. We make sure we order correctly in the 
file and we remove duplicates on process boundaries to avoid 
visualisation artefacts. With the HDF5 storage, we get orders of 
magnitude speeds-ups over VTK/XML for IO.

It might be possible to add markers, etc to XDMF in a consistent way. We 
haven't looked at that.

> It seems like you
> must have ran into some problems as IO for vectors was not supported
> for example.
> 

For IO of other objects, we bake our own HDF5 files. It would prefer to 
work against a specification, but that area is a big mess . . .

Garth

> Cheers
> Gerard
> 
> Begin forwarded message:
> 
>> FROM: "Gorman, Gerard J" <g.gorman at imperial.ac.uk>
>> 
>> SUBJECT: RE: [PETSC-DEV] DMPLEX READER / VIEWERS
>> 
>> DATE: 21 January 2014 23:58:11 GMT
>> 
>> TO: "Tim Tautges (ANL)" <tautges at mcs.anl.gov>
>> 
>> CC: petsc-dev <petsc-dev at mcs.anl.gov>
>> 
>> On 21 Jan 2014, at 21:07, Tim Tautges (ANL) <tautges at mcs.anl.gov>
>> wrote:
>> 
>> On 01/21/2014 12:00 PM, petsc-dev-request at mcs.anl.gov wrote:
>> 
>> Date: Tue, 21 Jan 2014 17:56:35 +0000
>> From: Blaise A Bourdin <bourdin at lsu.edu>
>> To: "Gorman, Gerard J" <g.gorman at imperial.ac.uk>
>> Cc: petsc-dev <petsc-dev at mcs.anl.gov>
>> Subject: Re: [petsc-dev] DMplex reader / viewers
>> Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu>
>> Content-Type: text/plain; charset="Windows-1252"
>> 
>> On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J
>> <g.gorman at imperial.ac.uk> wrote:
>> 
>> Hi Blaise
>> 
>> This is pretty much music to my ears. Interoperability is a major
>> headache, most file formats are rubbish (e.g. cannot support for
>> boundaries, limited meta data etc etc) most lack reasonable parallel
>> IO solutions (i.e. something other than 1 file per process)?and so
>> on. At the moment a few groups here at IC (this mostly means Michael
>> Lange ;) are integrating DMPlex with a range of FEM codes to try to
>> rationalise this. There are certainly some features missing (we?ve
>> bumped into most of that you?ve listed below) but all indications
>> are that DMPlex has the right design to be expanded to cover these
>> use cases.
>> 
>> On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:
>> 
>> Hi,
>> 
>> It looks like DMplex is steadily gaining maturity but I/O is
>> lagging behind. As far as I understand, right now, PETSc can _read_
>> a mesh in exodus format, and write binary VTS format, but many
>> issues remain, IMHO:
>> - The exodus reader relies on a hard-coded nodeset named ?marker?.
>> Generating such a nodeset is not trivial
>> (at least not for complex meshes generated with Cubit / Trelis).
>> - Reading from or writing to exodus files is not supported.
>> 
>> So we *really* want this as well for the purpose of dumping results
>> and checkpointing. Trying to shoehorn high order and DG data into
>> VTK is complete crap. I love VTK because its got loads of
>> functionality and it is easy to throw stuff together, but we are
>> getting ourselves into the position that we are just layering hack
>> after hack to deal with the fact that we cannot write all required
>> data into a vtk file. These days VTK/paraview has its own ExodusII
>> reader so we have a route to nice seamless integration.
>> 
>> - The VTS viewer only allows to read and write _all_ fields in a
>> DM. This may be overkill if one only
>> wants to read boundary values, for instance.
>> 
>> or only writing out prognostic fields for example.
>> 
>> - The VTS viewer loses all informations on exodus nodesets and cell
>> sets. These may have some significance
>> and may be required to exploit the output of a computations.
>> 
>> Right - this includes boundary labels, and it is just a fluff to
>> have to write this out into VTK. You would have to write a separate
>> vtu or something resulting in more messiness (and I already have
>> enough problems on LUSTER from having too many files).
>> 
>> - VTS seems to have a concept of ?blocks?. My understanding is that
>> the parallel VTS viewer uses blocks to
>> save subdomains, and that continuity of piecewise linear fields
>> across subdomain boundaries is lost.
>> It is not entirely clear to me if with this layout, it would be
>> possible to reopen a file with a
>> different processor count.
>> 
>> I think you just do not want to go there? For a start your vtk file
>> would not be a valid checkpoint to consider restarting on a
>> different number of processes. And secondly, it would just be a mess
>> to program.
>> 
>> I can dedicate some resources to improving DMplex I/O. Perhaps we
>> can start a discussion by listing the desired features such readers
>> / writers should have. I will pitch in by listing what matters to
>> me:
>> 
>> Keep talking?we have also an FTE working on this currently but this
>> is a long wish list and a lot of effort is required if this is to be
>> done within a reasonable time frame. It would be great if more
>> people were working on this.
>  I have a postdoc here who will devote some of his time to this task.
> 
>>> - A well documented and adopted file format that most
>>> post-processors / visualization tools can use
>> 
>> ExodusII appears to be the current favoured option.
>  Sadly yes? But SILO may be a close second and has a more modern
> interface.
> 
>>> - Ability to read / write individual fields
>>> - Preserve _all_ information from the exodus file (node / side /
>>> cell sets), do not lose continuity of fields
>>> across subdomain boundaries.
>>> - Ability to reopen file on a different cpu count
>> 
>> So this is where we need to have parallel IO support. Quoting from
>> petcs?s exodus.py
>> ??"
>> # ExodusII does not call HDF5 directly, but it does call
>> nc_def_var_deflate(), which is only
>> # part of libnetcdf when built using --enable-netcdf-4. Currently
>> --download-netcdf (netcdf.py)
>> # sets --enable-netcdf-4 only when HDF5 is enabled.
>> ??"
>> So, there may be some rebuilding required to ensure that all the
>> dependencies are built properly but it?s clearly there.
> 
>  I am not sure if Exodus has a good solution here. As far as I
> understand, exodus is inherently sequential, even when implemented
> with HDF5 instead of netcdf. I would also worry about third party
> support for exodus files using HDF5 as their storage format.
>  Exodus has an parallel extension called nemesis, but I can?t figure
> out how how their concept of ghost point and cells works. The
> documentation on this point is really unclear.
> 
>  I have to admit I was kind of hoping that ExodusII folks would have
> come on a bit more on the parallel IO front (I?m assuming those guys
> also run large simulations?). That said, I see this as a two stage
> process: first integrate with DMPlex as that should give the high
> level abstraction for read/write to file; secondly extend the family
> of readers/writers. At least this way there will be some agility and
> interoperability between different formats, and it will not be too
> disruptive to the application codes when a different formats adopted.
> 
>>> Considering the dismal state of parallel FE formats and libraries,
>>> it seems to me that one needs to chose between two options:
>>> 
>>> a. scatter back to a single I/O node and use sequential I/O using
>>> the ordering of the original (exodus) mesh. This allows reading
>>> and writing on an arbitrary number of processors, but has
>>> potential memory footprint and performance issues. How large a
>>> mesh can we reasonably expect to be able to handle this way?
> 
>  Personally I would stay far away from this option. Other than being a
> terrible serial bottleneck, it?s a major headache when you just want
> to run something just a little bit bigger than what happens to fit
> within a single node?
> 
>>> b. Do ?poor man? parallel I/O where each CPU does its own I/O, and
>>> possibly create interface matching files ? la nemesis or SILO.
>>> Maybe, we can save enough information on the parallel layout in
>>> order to easily write an un-partitionner as a post-processor.
> 
>  I am pretty sure that if we are writing everything in slabs to a HDF5
> container we do not have to worry too much about the parallel layout
> although some clear optimisations are possible. In the worst case it
> is a three stage process of where we perform a parallel read of the
> connectivity, scatter/gather for continuous numbering, parallel
> repartitioning and subsequent parallel read of remaining data.
> Importantly, it is at least scalable.
> 
>>> Unless one can come up with a better solution than a or b, I?d
>>> like to start by writing a very simple ASCII viewer demonstrating
>>> the communication involved, then modify it to use exodus, SILO or
>>> HDF5 format.
>>> 
>>> Blaise
> 
>>> 
>> 
>> Depending on the degree of direct interaction/automation in those
>> interactions between the mesh and Petsc, there are other options as
>> well. One that we're developing, based on the MOAB library, can
>> read/write (in serial) ExodusII, and also supports parallel
>> read/write using its own HDF5-based format. Parallel I/O robustness
>> has been iffy above ~16k procs and 32M-64M hex/tet elements, but for
>> smaller problems it should work. We're in the process of developing
>> direct support for going between a mesh defined with fields (called
>> tags in MOAB) and petsc vectors. MOAB has pretty solid support for
>> things like computing sharing and ghosting between procs and
>> exchanging/reducing field values on those entities. Viz is supported
>> either by compiling a VTK/Paraview plugin that pulls the mesh/fields
>> through MOAB or by translating to VTK (also supported directly from
>> MOAB); Visit also has a plugin you can enable. See
>> http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB [1] for details of
>> MOAB; the petsc integration stuff is on a bitbucket branch of petsc
>> owned by Vijay Mahadevan.
> 
>  Another reason this could be of great interest is that MOAB supports
> (according to the docs) geometric topology which could be used when
> adapting the mesh on a curved surface for example - another item on my
> which list. Is it integrated to PETSc via the plex or does this
> essentially replace the functionality of the plex? Why does it break
> down for more than 16k procs? is it just a case that Lustre gets
> hammered? What magic sauce is used by high order FEM codes such as
> nek500 that can run on ~1m cores?
> 
>> libmesh also maintains its own DMlibmesh, but I'm not sure how
>> solid their support for large mesh / parallel I/O is (but they've
>> been working on it recently I know).
> 
>  Are there any other formats that we should be considering? It?s a
> few years since I tried playing about with CGNS - at the time its
> parallel IO was non-existent and I have not seen it being pushed
> since. XDMF looks interesting as it is essentially some xml metadata
> and a HDF5 bucket. Is anyone championing this?
> 
>  Cheers
>  Gerard
> 
> 
> Links:
> ------
> [1] http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB
> [2]
> http://www.hector.ac.uk/cse/distributedcse/reports/UniDOLFIN/UniDOLFIN/index.html


From rupp at mcs.anl.gov  Wed Jan 22 06:19:57 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 13:19:57 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
Message-ID: <52DFB76D.3090601@mcs.anl.gov>

Hi again,

 > Well, I don't usually use PETSc  runtime options. I have only a 
subset of parameters (PC types, KSP types, etc.) which are interfaced by 
my solver and exposed to the end-user. Other few settings (matrix types 
for instance) are hardcoded in some objects (which my linear system 
solver interface makes use of).
>

Thanks for sending the hardcoded options, I could reproduce the problem 
with snes/examples/tutorials/ex2.c using the runtime options
  -pc_type asm -vec_type cusp -mat_type aijcusp
Actually, the problem can be reduced down to only providing
  -vec_type cusp

It seems to be the same problem as the one described here:
   http://lists.mcs.anl.gov/pipermail/petsc-dev/2014-January/014460.html
which I couldn't track down yet. I'll let you know as soon as this is fixed.

Best regards,
Karli



From andrea.lani at gmail.com  Wed Jan 22 06:25:11 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Wed, 22 Jan 2014 13:25:11 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <52DFB76D.3090601@mcs.anl.gov>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
	<52DFB76D.3090601@mcs.anl.gov>
Message-ID: <A0110DF2-FE3A-4A01-9007-7AB4EDD5E92F@gmail.com>

Great! I'm happy not to be the only one having the problem 

On Jan 22, 2014, at 1:19 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi again,
> 
> > Well, I don't usually use PETSc  runtime options. I have only a subset of parameters (PC types, KSP types, etc.) which are interfaced by my solver and exposed to the end-user. Other few settings (matrix types for instance) are hardcoded in some objects (which my linear system solver interface makes use of).
>> 
> 
> Thanks for sending the hardcoded options, I could reproduce the problem with snes/examples/tutorials/ex2.c using the runtime options
> -pc_type asm -vec_type cusp -mat_type aijcusp
> Actually, the problem can be reduced down to only providing
> -vec_type cusp
> 
> It seems to be the same problem as the one described here:
>  http://lists.mcs.anl.gov/pipermail/petsc-dev/2014-January/014460.html
> which I couldn't track down yet. I'll let you know as soon as this is fixed.
> 
> Best regards,
> Karli
> 


From rupp at mcs.anl.gov  Wed Jan 22 06:34:39 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 13:34:39 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <A0110DF2-FE3A-4A01-9007-7AB4EDD5E92F@gmail.com>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
	<52DFB76D.3090601@mcs.anl.gov>
	<A0110DF2-FE3A-4A01-9007-7AB4EDD5E92F@gmail.com>
Message-ID: <52DFBADF.2010309@mcs.anl.gov>


> Great! I'm happy not to be the only one having the problem

Well, I'm *not* happy that users have this problem... ;-)

Best regards,
Karli



From andrea.lani at gmail.com  Wed Jan 22 06:46:42 2014
From: andrea.lani at gmail.com (Andrea Lani)
Date: Wed, 22 Jan 2014 13:46:42 +0100
Subject: [petsc-dev] PCASM+KSPGMRES
In-Reply-To: <52DFBADF.2010309@mcs.anl.gov>
References: <CALjjKKXLTLX0VDMg0jdixuKcSrv6pSBO2hvUytYSLiMtAAxeVA@mail.gmail.com>
	<52DEE8CB.8060202@mcs.anl.gov>
	<5E8214F7-C376-489E-B62D-E2F117AC09C2@gmail.com>
	<52DFB76D.3090601@mcs.anl.gov>
	<A0110DF2-FE3A-4A01-9007-7AB4EDD5E92F@gmail.com>
	<52DFBADF.2010309@mcs.anl.gov>
Message-ID: <65A39B3E-16F3-4D13-8110-FD595FB91F04@gmail.com>

Sure, but we have to keep you busy with challenging problems, somehow ;-)

On Jan 22, 2014, at 1:34 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> 
>> Great! I'm happy not to be the only one having the problem
> 
> Well, I'm *not* happy that users have this problem... ;-)
> 
> Best regards,
> Karli
> 


From jed at jedbrown.org  Mon Jan 20 14:32:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 20 Jan 2014 13:32:58 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <m2y52abdqu.fsf@gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<m2y52abdqu.fsf@gmail.com>
Message-ID: <877g9ul7gl.fsf@jedbrown.org>

Sean Farley <sean.michael.farley at gmail.com> writes:
> There would have to a configurable subprefix, e.g. include/petsc-3.4,
> lib/petsc-3.4, etc. Though, it's not as big a deal to me.

If we include versions, people could put this in their makefiles:

CFLAGS := $(shell petsc-3.5-config --cflags)
LIBS := $(shell petsc-3.5-config --libs)


and users would not have to think about paths.  This would allow
distributions (and us) to ship binaries that install next to each other.
It's not the organization typically used on clusters, but it seems to me
that it would make distribution easier on personal/small machines.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/dc9d929b/attachment.sig>

From jed at jedbrown.org  Mon Jan 20 13:53:31 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 20 Jan 2014 12:53:31 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
Message-ID: <87bnz6l9ac.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>> Jed's suggestions here would be great for package maintainers. As an
>> added bonus, it'd be great if $prefix/share/petsc-$foo could be be
>> configurable (so that multiple installations could be live
>> side-by-side).
>
>     This would allow multiple copies of the ?share? stuff but if the
>     library is in $prefix/lib/libpetsc.dylib how can multiple
>     installations exist side by side?

If we decide that a petsc-config is the canonical way to "find" PETSc,
we could do like Python (python-config, python2-config,
python3.3-config, etc).  Should we install

  $prefix/bin/petsc-${version}-${arch}-config

and symlink $prefix/bin/petsc-config to it by default?

Then always namespace libraries and includes?

I'm not sure what to do on this last point.  I would like to be able to
install multiple versions to the same prefix (especially debug+opt), but
it also seems weird to not support -lpetsc (instead -lpetsc-3.5-mpich-opt).

What do you think?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140120/d242c912/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 10:07:38 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 09:07:38 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
Message-ID: <87mwipiaid.fsf@jedbrown.org>

Blaise A Bourdin <bourdin at lsu.edu> writes:

> Hi,
>
> It looks like DMplex is steadily gaining maturity but I/O is lagging
> behind. As far as I understand, right now, PETSc can _read_ a mesh in
> exodus format, and write binary VTS format, but many issues remain,
> IMHO:
>    - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>      (at least not for complex meshes generated with Cubit / Trelis).

Agreed.

>    - Reading from or writing to exodus files is not supported.

Reading works.  (typo?)

>    - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only 
>      wants to read boundary values, for instance.

Would it be okay to create a sub-DM containing the boundary fields and
write that Vec?

>    - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>      and may be required to exploit the output of a computations.

I consider VTS to be quick-and-dirty, but not a proper solution.

>    - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>      save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost. 
>      It is not entirely clear to me if with this layout, it would be possible to reopen a file with a 
>      different processor count.

I don't use PVTS (write many files and an index file to stitch them
together).  Instead, I write one file containing multiple blocks.  They
should be consistent at interfaces.

> I can dedicate some resources to improving DMplex I/O. Perhaps we can
> start a discussion by listing the desired features such readers /
> writers should have. I will pitch in by listing what matters to me:
>    - A well documented and adopted file format that most post-processors / visualization tools can use

Suggestions?  I haven't met a file format that wasn't garbage, though
some are less trashy than others...

>    - Ability to read / write individual fields

I prefer sub-DM/subvecs.  If you don't like that, can you suggest an
alternative interface?

>    - Preserve _all_ information from the exodus file (node / side /
>    cell sets), 

Seems attainable for everything that DMPlex can encode.

>    do not lose continuity of fields across subdomain boundaries.

I think this is correct now.

>    - Ability to reopen file on a different cpu count
>    - Support for higher order elements

Do you know a format that does this?  I couldn't find one and thus wrote
my own HDF5 format and VisIt plugin.  We could do that for DMPlex, but
it's something that people have to install.  Using common formats
(especially VTS) is easy.

> Am I missing something? If not, we can follow up with discussion on formats and implementation.

One attribute I didn't like about my format was that my data model was
more relational than hierarchical, for which queries are cumbersome to
write in HDF5.  I think the normalized relational model is valuable, so
I would consider instead doing a sqlite index backed by "dumb" HDF5 or
even MPI-IO objects.


Do you include material properties in mesh files?  This is common with
systems like Abaqus, and has been a workflow impediment for us in the
past.  I'd be happy to hold that information outside the mesh file, but
it needs to be represented and associated somehow.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/cf18b5df/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 11:20:36 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 10:20:36 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
Message-ID: <87a9epi74r.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>    - Reading from or writing to exodus files is not supported.
>>
>
> Yes, I think this is the best target. It should be similar to writing HDF5
> that we do for PyLith.

How are we going to represent high-order elements in Exodus?  Doesn't it
only support quadratic continuous spaces?  What about DG spaces, H(div)
spaces, or higher than second order?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/e52c5425/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 12:04:32 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 11:04:32 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <ECA3EBD3-7618-490A-B167-B562428ECC7B@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<EE838CA3-FBE7-4CEF-A67F-354F14C7629F@imperial.ac.uk>
	<ECA3EBD3-7618-490A-B167-B562428ECC7B@lsu.edu>
Message-ID: <87txcxgqj3.fsf@jedbrown.org>

Blaise A Bourdin <bourdin at lsu.edu> writes:
> I am not sure if Exodus has a good solution here. As far as I
> understand, exodus is inherently sequential, even when implemented
> with HDF5 instead of netcdf. I would also worry about third party
> support for exodus files using HDF5 as their storage format.  Exodus
> has an parallel extension called nemesis, but I can?t figure out how
> how their concept of ghost point and cells works. The documentation on
> this point is really unclear.

PyLith accesses Exodus files directly via the NetCDF interface rather
than through the Exodus library.  I don't recall if it has true parallel
IO that way, but it's certainly possible.

> Considering the dismal state of parallel FE formats and libraries, it
> seems to me that one needs to chose between two options:
>
>  a. scatter back to a single I/O node and use sequential I/O using the
>  ordering of the original (exodus) mesh. This allows reading and
>  writing on an arbitrary number of processors, but has potential
>  memory footprint and performance issues. How large a mesh can we
>  reasonably expect to be able to handle this way?
>
>  b. Do ?poor man? parallel I/O where each CPU does its own I/O, and

This "independent" IO leads to atrocious performance on parallel file
systems.  You can manually aggregate to a smaller number of processes
and have those write, but collective IO is simpler and more sustainable.

>  possibly create interface matching files ? la nemesis or SILO. Maybe,
>  we can save enough information on the parallel layout in order to
>  easily write an un-partitionner as a post-processor.
>
> Unless one can come up with a better solution than a or b, I?d like to
> start by writing a very simple ASCII viewer demonstrating the
> communication involved, 

The VTS viewer is mode (a), but could simply be extended to support
writing all the heavy data using MPI-IO collectives.

> then modify it to use exodus, SILO or HDF5 format.

Is SILO expressive enough for high-order, DG, H(div), etc?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/caca3716/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 23:01:40 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 22:01:40 -0700
Subject: [petsc-dev] How do I build a shared library for exodus?
In-Reply-To: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
References: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
Message-ID: <87lhy8haob.fsf@jedbrown.org>

Blaise A Bourdin <bourdin at lsu.edu> writes:

> Hi,
>
> For some reason, I would need to generate a shared library for
> exodusII, but the Makefile included in the source distribution does
> not have a rule for this. Is there an easy way to modify the build
> system package in order to generate the shared library?

The Makefile.standalone does not have rules for shared.  Perhaps
upstream exodus (now at version 6.02) is recommending the CMake build?
I'd rather not have exodusii.py depend on CMake if we can help it
(buggy, bad logging, another thing to compile), so maybe we need to add
a shared rule?  Or embrace the beast?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/b17332bb/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 23:23:55 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 22:23:55 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
	<52DEE19A.30306@mcs.anl.gov>
	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
Message-ID: <87iotch9n8.fsf@jedbrown.org>

"Gorman, Gerard J" <g.gorman at imperial.ac.uk> writes:
>>>  a. scatter back to a single I/O node and use sequential I/O using
>>>  the ordering of the original (exodus) mesh. This allows reading and
>>>  writing on an arbitrary number of processors, but has potential
>>>  memory footprint and performance issues. How large a mesh can we
>>>  reasonably expect to be able to handle this way?
>
> Personally I would stay far away from this option. Other than being a
> terrible serial bottleneck, it?s a major headache when you just want
> to run something just a little bit bigger than what happens to fit
> within a single node?

The VTU viewer does it in a memory-scalable way.  The only reason to do
this is to interact with a file format to which we cannot do parallel
writes.  For VTU (XML with binary-appended), we could use MPI-IO for the
binary appended data.  The Exodus library (formerly separate in Nemesis)
does pre-decomposed meshes which is a crazy workflow.  We want mesh
files to be independent of the number of processors; we'll read in
parallel and partition on the fly.  I believe we can do this by using
the NetCDF-4 interface to access the Exodus file directly.

>>>  b. Do ?poor man? parallel I/O where each CPU does its own I/O, and
>>>  possibly create interface matching files ? la nemesis or
>>>  SILO. Maybe, we can save enough information on the parallel layout
>>>  in order to easily write an un-partitionner as a post-processor.
>
> I am pretty sure that if we are writing everything in slabs to a HDF5
> container we do not have to worry too much about the parallel layout
> although some clear optimisations are possible. In the worst case it
> is a three stage process of where we perform a parallel read of the
> connectivity, scatter/gather for continuous numbering, parallel
> repartitioning and subsequent parallel read of remaining
> data. Importantly, it is at least scalable.

Yeah, though vis doesn't like collective interfaces let alone
partitioning on the fly, so we'll either need our own reader plugin or
to play some games.

> What magic sauce is used by high order FEM codes such as
> nek500 that can run on ~1m cores?

Nek runs about one element per core so its challenges are different.

> Are there any other formats that we should be considering? It?s a few
> years since I tried playing about with CGNS - at the time its parallel
> IO was non-existent and I have not seen it being pushed since. 

We have (crude) CGNS support in PETSc.  It's an HDF5 format.  I think
it's comparable to Exodus, but Cubit doesn't write it natively.
(Meanwhile, lots of other meshing tools don't write Exodus.)

> XDMF looks interesting as it is essentially some xml metadata and a
> HDF5 bucket. 

It feels like VTK to me, except that you can put the arrays in an HDF5
file instead of appending them to the XML file.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/3f4f3979/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 12:04:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 11:04:58 -0700
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpd-E02zyFd2rVgeOf9KaTTiRS3agaPhqqbZGM5p1xZG0Q@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
	<8738kjqy3c.fsf@jedbrown.org>
	<CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>
	<87mwirpiq3.fsf@jedbrown.org>
	<CAJ1ofpd-E02zyFd2rVgeOf9KaTTiRS3agaPhqqbZGM5p1xZG0Q@mail.gmail.com>
Message-ID: <87r481gqid.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:

> On Sun, Jan 19, 2014 at 5:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>> Geoffrey Irving <irving at naml.us> writes:
>>> It isn't, the only difference is that the function I'm imagining needs
>>> an extra quadrature rule argument.  Nothing here is complicated, so if
>>> there are no particular opinions I'll write the function and send a
>>> patch.  Can I write it hard coded to MPI_SUM first, and we can discuss
>>> how to generalize it after that?  I am less clear on how to best write
>>> the generalized version.
>>
>> That's fine with me.
>
> Should the result be an array of PetscScalar or PetscReal?  My
> application is for objectives, so PetscReal, but PetscScalar is more
> generic.

I would use PetscScalar.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/d3c4bff5/attachment.sig>

From jed at jedbrown.org  Tue Jan 21 23:52:34 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 21 Jan 2014 22:52:34 -0700
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
Message-ID: <874n4wh8bh.fsf@jedbrown.org>

Lisandro Dalcin <dalcinl at gmail.com> writes:
> I thought to do that in PetIGA, but then realized that a reducer+eval
> at quadrature points+MPI_MAX sounds weird (though I can imagine some
> use cases). A reduce with MPI_MAX is a lucky consequence of computing
> integrals through quadrature.

Suppose we want to know the max stress in a structure to determine
whether it will be damaged by a given loading scenario?  The quadrature
points are not sufficient to determine a max stress for the discrete
solution, but it is a convergent method for determining the max stress
of the continuum structure.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140121/978d64c2/attachment.sig>

From sean.michael.farley at gmail.com  Wed Jan 22 10:19:46 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Wed, 22 Jan 2014 10:19:46 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87bnz6l9ac.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org>
	<84121E5B-38BF-4B39-9D53-534908B6AC48@mcs.anl.gov>
	<878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org>
Message-ID: <m2eh40at0d.fsf@gmail.com>


jed at jedbrown.org writes:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>> Jed's suggestions here would be great for package maintainers. As an
>>> added bonus, it'd be great if $prefix/share/petsc-$foo could be be
>>> configurable (so that multiple installations could be live
>>> side-by-side).
>>
>>     This would allow multiple copies of the ?share? stuff but if the
>>     library is in $prefix/lib/libpetsc.dylib how can multiple
>>     installations exist side by side?
>
> If we decide that a petsc-config is the canonical way to "find" PETSc,
> we could do like Python (python-config, python2-config,
> python3.3-config, etc).  Should we install
>
>   $prefix/bin/petsc-${version}-${arch}-config
>
> and symlink $prefix/bin/petsc-config to it by default?
>
> Then always namespace libraries and includes?

This is what I had in mind as well. Though, distributions would remove
the symlink since that would conflict.

> I'm not sure what to do on this last point.  I would like to be able to
> install multiple versions to the same prefix (especially debug+opt), but
> it also seems weird to not support -lpetsc (instead -lpetsc-3.5-mpich-opt).
>
> What do you think?

I'm sure you're talking to Barry here but I'll chime in anyway :-) I
have rarely been in a situation where I am compiling PETSc without a
crapton of other libraries, so saving the extra typing of -lpetsc
vs. -lpetsc-3.5-mpich-opt is almost meaningless.


From dmeiser at txcorp.com  Wed Jan 22 10:49:00 2014
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Wed, 22 Jan 2014 09:49:00 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
Message-ID: <52DFF67C.5000504@txcorp.com>

Hi,

I'm trying to understand VecScatterInitializeForGPU in 
src/vec/vec/utils/veccusp/vscatcusp.c. I don't understand why this 
function can get away with casting the fromdata and todata in the inctx 
to VecScatter_MPI_General. Don't we need to inspect the VecScatterType 
fields of the todata and fromdata?

Cheers,
Dominic

-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com



From jed at jedbrown.org  Wed Jan 22 10:48:12 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 09:48:12 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <m2eh40at0d.fsf@gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org> <878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
Message-ID: <87wqhsdktv.fsf@jedbrown.org>

Sean Farley <sean.michael.farley at gmail.com> writes:
>>   $prefix/bin/petsc-${version}-${arch}-config
>>
>> and symlink $prefix/bin/petsc-config to it by default?
>>
>> Then always namespace libraries and includes?
>
> This is what I had in mind as well. Though, distributions would remove
> the symlink since that would conflict.

Debian would probably manage it through /etc/alternatives.

>> I'm not sure what to do on this last point.  I would like to be able to
>> install multiple versions to the same prefix (especially debug+opt), but
>> it also seems weird to not support -lpetsc (instead -lpetsc-3.5-mpich-opt).
>>
>> What do you think?
>
> I'm sure you're talking to Barry here but I'll chime in anyway :-) I
> have rarely been in a situation where I am compiling PETSc without a
> crapton of other libraries, so saving the extra typing of -lpetsc
> vs. -lpetsc-3.5-mpich-opt is almost meaningless.

Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/73904d6b/attachment.sig>

From jed at jedbrown.org  Wed Jan 22 10:57:41 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 09:57:41 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
Message-ID: <87iotcdke2.fsf@jedbrown.org>

Moving back to list.

Blaise A Bourdin <bourdin at lsu.edu> writes:

> On Jan 21, 2014, at 11:20 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>>>   - Reading from or writing to exodus files is not supported.
>>>> 
>>> 
>>> Yes, I think this is the best target. It should be similar to writing HDF5
>>> that we do for PyLith.
>> 
>> How are we going to represent high-order elements in Exodus?  Doesn't it
>> only support quadratic continuous spaces?  What about DG spaces, H(div)
>> spaces, or higher than second order?
>
> It only supports first and second order elements. I don?t think that continuity is an issue. Nothing prevents you to alter the connectivity table.
> Element types is set by blocks, so mixing polynomial orders is not possible without some trickery

Do we consider this an acceptable long-term solution?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/2eb7a092/attachment.sig>

From paulmullowney at gmail.com  Wed Jan 22 11:05:13 2014
From: paulmullowney at gmail.com (Paul Mullowney)
Date: Wed, 22 Jan 2014 10:05:13 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <52DFF67C.5000504@txcorp.com>
References: <52DFF67C.5000504@txcorp.com>
Message-ID: <CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>

Dominic,

A few years ago, I was trying to minimize the amount of data transfer to
and from the GPU (for multi-GPU MatMult) by inspecting the indices of the
data that needed to be message to and from the device. Then, I would call
gather kernels on the GPU which pulled the scattered data into contiguous
buffers and then be transferred to the host asynchronously (while the
MatMult was occurring). The existence of VecScatterInitializeForGPU was
added in order to build the necessary buffers as needed. This was the
motivation behind the existence of VecScatterInitializeForGPU.

An alternative approach is to message the smallest contiguous buffer
containing all the data with a single cudaMemcpyAsync. This is the method
currently implemented.

I never found a case where the former implementation (with a GPU
gather-kernel) performed better than the alternative approach which
messaged the smallest contiguous buffer. I looked at many, many matrices.

Now, as far as I understand the VecScatter kernels, this method should only
get called if the transfer is MPI_General (i.e. PtoP parallel to parallel).
Other VecScatter methods are called in other circumstances where the the
scatter is not MPI_General. That assumption could be wrong though.

-Paul




On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:

> Hi,
>
> I'm trying to understand VecScatterInitializeForGPU in
> src/vec/vec/utils/veccusp/vscatcusp.c. I don't understand why this
> function can get away with casting the fromdata and todata in the inctx to
> VecScatter_MPI_General. Don't we need to inspect the VecScatterType fields
> of the todata and fromdata?
>
> Cheers,
> Dominic
>
> --
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/8c099db1/attachment.html>

From baagaard at usgs.gov  Wed Jan 22 11:16:20 2014
From: baagaard at usgs.gov (Brad Aagaard)
Date: Wed, 22 Jan 2014 09:16:20 -0800
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <87iotcdke2.fsf@jedbrown.org>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>	<87a9epi74r.fsf@jedbrown.org>	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org>
Message-ID: <52DFFCE4.1020503@usgs.gov>

On 01/22/2014 08:57 AM, Jed Brown wrote:
> Moving back to list.
>
> Blaise A Bourdin <bourdin at lsu.edu> writes:
>
>> On Jan 21, 2014, at 11:20 AM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>    - Reading from or writing to exodus files is not supported.
>>>>>
>>>>
>>>> Yes, I think this is the best target. It should be similar to writing HDF5
>>>> that we do for PyLith.
>>>
>>> How are we going to represent high-order elements in Exodus?  Doesn't it
>>> only support quadratic continuous spaces?  What about DG spaces, H(div)
>>> spaces, or higher than second order?
>>
>> It only supports first and second order elements. I don?t think that continuity is an issue. Nothing prevents you to alter the connectivity table.
>> Element types is set by blocks, so mixing polynomial orders is not possible without some trickery
>
> Do we consider this an acceptable long-term solution?
>

I see a DMPlex layout for HDF-5 files as a convenient means of providing 
fast, parallel I/O with Xdmf files as a means to provide a way for 
ParaView/Visit, etc to be able to display much of the info within the 
HDF5-files without custom plug-ins. Plug-ins might be necessary for 
specialized information that can't be expressed in Xdmf files. We deal 
with end-users with limited programming experience for which plug-ins 
might be difficult.

For PyLith, I like being able to use h5py for high-level access to the 
info in the HDF5 files for post processing.

We have plans to add higher order basis functions to PyLith this spring, 
and recognize the deficiencies in many of the available formats. It 
would be nice to have a consistent way to deal with DMPlex I/O and these 
other issues (boundaries, high order, etc) across various codes using 
DMPlex.

Brad



From bourdin at lsu.edu  Wed Jan 22 11:21:31 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Wed, 22 Jan 2014 17:21:31 +0000
Subject: [petsc-dev] How do I build a shared library for exodus?
In-Reply-To: <87lhy8haob.fsf@jedbrown.org>
References: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
	<87lhy8haob.fsf@jedbrown.org>
Message-ID: <28EB76F8-25FB-45D4-B482-543101E7A0D8@lsu.edu>


On Jan 21, 2014, at 11:01 PM, Jed Brown <jed at jedbrown.org> wrote:

> Blaise A Bourdin <bourdin at lsu.edu> writes:
> 
>> Hi,
>> 
>> For some reason, I would need to generate a shared library for
>> exodusII, but the Makefile included in the source distribution does
>> not have a rule for this. Is there an easy way to modify the build
>> system package in order to generate the shared library?
> 
> The Makefile.standalone does not have rules for shared.  Perhaps
> upstream exodus (now at version 6.02) is recommending the CMake build?
> I'd rather not have exodusii.py depend on CMake if we can help it
> (buggy, bad logging, another thing to compile), so maybe we need to add
> a shared rule?  Or embrace the beast?

let?s say we do want to mess with cmake. What is the typical way to add the shared rule?
   - patch Makefile.standalone in exodusii.py?
   - repackage exodus and serve it from the petsc servers?
Is there a BuildSystem way to know that the linker flag to build a shared library is -shared under linux and -dynamiclib under mac OS? If so, can we simply build the shared library in exodusii.py?

Blaise 
-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From sean.michael.farley at gmail.com  Wed Jan 22 11:22:26 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Wed, 22 Jan 2014 11:22:26 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87wqhsdktv.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org> <878uue189s.fsf@jedbrown.org>
	<E6D173A5-C277-42F3-ADE6-E5642EC5E0BD@mcs.anl.gov>
	<871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org>
Message-ID: <m2d2jkaq3x.fsf@gmail.com>


jed at jedbrown.org writes:

> Sean Farley <sean.michael.farley at gmail.com> writes:
>>>   $prefix/bin/petsc-${version}-${arch}-config
>>>
>>> and symlink $prefix/bin/petsc-config to it by default?
>>>
>>> Then always namespace libraries and includes?
>>
>> This is what I had in mind as well. Though, distributions would remove
>> the symlink since that would conflict.
>
> Debian would probably manage it through /etc/alternatives.

Just a note that MacPorts would manage it through etc/select :-)


From bourdin at lsu.edu  Wed Jan 22 11:26:02 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Wed, 22 Jan 2014 17:26:02 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <87iotcdke2.fsf@jedbrown.org>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org>
Message-ID: <6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9@lsu.edu>


On Jan 22, 2014, at 10:57 AM, Jed Brown <jed at jedbrown.org> wrote:

> Moving back to list.
sorry, muscle memory prevents me from hitting that reply-to button

> 
> Blaise A Bourdin <bourdin at lsu.edu> writes:
> 
>> On Jan 21, 2014, at 11:20 AM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>  - Reading from or writing to exodus files is not supported.
>>>>> 
>>>> 
>>>> Yes, I think this is the best target. It should be similar to writing HDF5
>>>> that we do for PyLith.
>>> 
>>> How are we going to represent high-order elements in Exodus?  Doesn't it
>>> only support quadratic continuous spaces?  What about DG spaces, H(div)
>>> spaces, or higher than second order?
>> 
>> It only supports first and second order elements. I don?t think that continuity is an issue. Nothing prevents you to alter the connectivity table.
>> Element types is set by blocks, so mixing polynomial orders is not possible without some trickery
> 
> Do we consider this an acceptable long-term solution?

hell no!

Does anybody knows of any open and widespread format supporting elements of order >2? I checked quickly, and according to their documentation it seems that none of exodusii, silo 4.7, or xdmf support them...

Incidentally, exodus docs last update is from 2006,silo?s from 2010, and http://www.xdmf.org has seen minor edits after 2009... I seem to recall that the project is now hosted with visit. Not sure.

Blaise


-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From mc0710 at gmail.com  Wed Jan 22 11:26:25 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Wed, 22 Jan 2014 11:26:25 -0600
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <52DEE695.5080200@mcs.anl.gov>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>
	<52DE3DBC.80000@mcs.anl.gov>
	<CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>
	<52DEE695.5080200@mcs.anl.gov>
Message-ID: <CAJzU6sCEDeyHFnTKU4zc+zOhqpbL77GvKzucVCwTOeGAFaHyJg@mail.gmail.com>

Thanks Karl. Is there anyway I can get the build info?

Also, my system is such that I have the GPU on one platform (0) and the CPU
on another (1). When I try using -viennalcl_device_cpu, it still uses the
GPU cause I think that it defaults to the first platform it finds which
only has the GPU. Is there some way to toggle between the platforms or do
you suggest that I manually pass in the contexts into viennacl?

Cheers,
Mani


On Tue, Jan 21, 2014 at 3:28 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > I like to pass in "-cl-nv-verbose" for compilation on nvidia cards.
>
> ok, in such case you also want access to the compiler output, don't you?
>
>
>  Also, I pass in parameters, for ex "-D Nx=128 -D Ny=128". I'll look at
>> the ViennaCL api.
>>
>
> Use
>   viennacl::ocl::current_context().build_options("flags_here");
> and make sure you have VIENNACL_WITH_OPENCL defined for your compilation
> unit. I'll extend the PETSc API suitably when updating the ViennaCL
> bindings to the latest 1.5.1 release.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/d9a5f44b/attachment.html>

From balay at mcs.anl.gov  Wed Jan 22 11:28:57 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 22 Jan 2014 11:28:57 -0600
Subject: [petsc-dev] How do I build a shared library for exodus?
In-Reply-To: <28EB76F8-25FB-45D4-B482-543101E7A0D8@lsu.edu>
References: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
	<87lhy8haob.fsf@jedbrown.org>
	<28EB76F8-25FB-45D4-B482-543101E7A0D8@lsu.edu>
Message-ID: <alpine.LFD.2.11.1401221123110.2367@asterix>

On Wed, 22 Jan 2014, Blaise A Bourdin wrote:

> 
> On Jan 21, 2014, at 11:01 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> > Blaise A Bourdin <bourdin at lsu.edu> writes:
> > 
> >> Hi,
> >> 
> >> For some reason, I would need to generate a shared library for
> >> exodusII, but the Makefile included in the source distribution does
> >> not have a rule for this. Is there an easy way to modify the build
> >> system package in order to generate the shared library?
> > 
> > The Makefile.standalone does not have rules for shared.  Perhaps
> > upstream exodus (now at version 6.02) is recommending the CMake build?

If upstream recommends cmake - perhaps we should stick to that? We
already have a bunch of packages that require cmake.

balay at asterix /home/balay/petsc (master)
$ git grep framework.require |grep PETSc.packages.cmake
config/PETSc/packages/clique.py:    self.cmake           = framework.require('PETSc.packages.cmake',self)
config/PETSc/packages/elemental.py:    self.cmake           = framework.require('PETSc.packages.cmake',self)
config/PETSc/packages/metis.py:    self.cmake           = framework.require('PETSc.packages.cmake',self)
config/PETSc/packages/parmetis.py:    self.cmake           = framework.require('PETSc.packages.cmake',self)
config/cmakeboot.py:   self.cmake         = self.framework.require('PETSc.packages.cmake',       None)


> > I'd rather not have exodusii.py depend on CMake if we can help it
> > (buggy, bad logging, another thing to compile), so maybe we need to add
> > a shared rule?  Or embrace the beast?
> 
> let?s say we do want to mess with cmake. What is the typical way to add the shared rule?
>    - patch Makefile.standalone in exodusii.py?
>    - repackage exodus and serve it from the petsc servers?

Perhaps upstream exodus tarball should support sharedlibrary/static
library as a build option? elemental.py appears to have the followig
code commented..

if self.sharedLibraries.useShared:
  args.append('-DSHARE_LIBRARIES=ON')          

Satish

> Is there a BuildSystem way to know that the linker flag to build a shared library is -shared under linux and -dynamiclib under mac OS? If so, can we simply build the shared library in exodusii.py?
> 
> Blaise 
> 

From dmeiser at txcorp.com  Wed Jan 22 11:32:08 2014
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Wed, 22 Jan 2014 10:32:08 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
References: <52DFF67C.5000504@txcorp.com>
	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
Message-ID: <52E00098.4050601@txcorp.com>

Hey Paul,

Thanks for providing background on this.

On Wed 22 Jan 2014 10:05:13 AM MST, Paul Mullowney wrote:
>
> Dominic,
> A few years ago, I was trying to minimize the amount of data transfer
> to and from the GPU (for multi-GPU MatMult) by inspecting the indices
> of the data that needed to be message to and from the device. Then, I
> would call gather kernels on the GPU which pulled the scattered data
> into contiguous buffers and then be transferred to the host
> asynchronously (while the MatMult was occurring). The existence of
> VecScatterInitializeForGPU was added in order to build the necessary
> buffers as needed. This was the motivation behind the existence of
> VecScatterInitializeForGPU.
> An alternative approach is to message the smallest contiguous buffer
> containing all the data with a single cudaMemcpyAsync. This is the
> method currently implemented.
> I never found a case where the former implementation (with a GPU
> gather-kernel) performed better than the alternative approach which
> messaged the smallest contiguous buffer. I looked at many, many matrices.
> Now, as far as I understand the VecScatter kernels, this method should
> only get called if the transfer is MPI_General (i.e. PtoP parallel to
> parallel). Other VecScatter methods are called in other circumstances
> where the the scatter is not MPI_General. That assumption could be
> wrong though.


I see. I figured there was some logic in place to make sure that this 
function only gets called in cases where the transfer type is 
MPI_General. I'm getting segfaults in this function where the todata and 
fromdata are of a different type. This could easily be user error but 
I'm not sure. Here is an example valgrind error:

==27781== Invalid read of size 8
==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*) 
(mpiaijcusparse.cu:108)
==27781== by 0xA20CC3: MatMult (matrix.c:2242)
==27781== by 0x4645E4: main (ex7.c:93)
==27781== Address 0x286305e0 is 1,616 bytes inside a block of size 1,620 
alloc'd
==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*, 
char const*, void**) (mal.c:27)
==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char 
const*, char const*, void**) (mtr.c:186)
==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType) 
(mpiaij.c:706)
==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
==27781== by 0x464301: main (ex7.c:78)

This was produced by src/ksp/ksp/tutorials/ex7.c. The command line 
options are

./ex7 -mat_type mpiaijcusparse -vec_type cusp

In this particular case the todata is of type VecScatter_Seq_Stride and 
fromdata is of type VecScatter_Seq_General. The complete valgrind log 
(including configure options for petsc) is attached.

Any comments or suggestions are appreciated.
Cheers,
Dominic

>
> -Paul
>
>
> On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser <dmeiser at txcorp.com
> <mailto:dmeiser at txcorp.com>> wrote:
>
> Hi,
>
> I'm trying to understand VecScatterInitializeForGPU in
> src/vec/vec/utils/veccusp/__vscatcusp.c. I don't understand why
> this function can get away with casting the fromdata and todata in
> the inctx to VecScatter_MPI_General. Don't we need to inspect the
> VecScatterType fields of the todata and fromdata?
>
> Cheers,
> Dominic
>
> -- 
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036 <tel:303-996-2036>
> Fax: 303-448-7756 <tel:303-448-7756>
> www.txcorp.com <http://www.txcorp.com>
>
>



-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com

-------------- next part --------------
==27786== Memcheck, a memory error detector
==27786== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==27786== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==27786== Command: ./ex7 -mat_type mpiaijcusparse -vec_type cusp
==27786== 
==27786== Syscall param writev(vector[...]) points to uninitialised byte(s)
==27786==    at 0x157E433B: writev (in /lib64/libc-2.12.so)
==27786==    by 0x1490D806: mca_oob_tcp_msg_send_handler (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x1490E82C: mca_oob_tcp_peer_send (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x14910BEC: mca_oob_tcp_send_nb (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x1492A315: orte_rml_oob_send (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x1492A55F: orte_rml_oob_send_buffer (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x148F8327: modex (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x147E7ACA: ompi_mpi_init (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x147FE57F: PMPI_Init_thread (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x4929D9: PetscInitialize (pinit.c:777)
==27786==    by 0x463B11: main (ex7.c:49)
==27786==  Address 0x16c8efc1 is 161 bytes inside a block of size 256 alloc'd
==27786==    at 0x4C27BE0: realloc (vg_replace_malloc.c:662)
==27786==    by 0x14943AC2: opal_dss_buffer_extend (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x14943C84: opal_dss_copy_payload (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x148F11D6: orte_grpcomm_base_pack_modex_entries (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x148F82DC: modex (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x147E7ACA: ompi_mpi_init (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x147FE57F: PMPI_Init_thread (in /scr_ivy/dmeiser/PTSOLVE/openmpi-1.6.1-nodl/lib/libmpi.so.1.0.3)
==27786==    by 0x4929D9: PetscInitialize (pinit.c:777)
==27786==    by 0x463B11: main (ex7.c:49)
==27786== 
==27786== Warning: set address range perms: large range [0x800000000, 0x1100000000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x3aeed000, 0x33aeed000) (noaccess)
==27786== Warning: set address range perms: large range [0x1100000000, 0x1400000000) (noaccess)
==27786== Invalid read of size 8
==27786==    at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
==27786==    by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*) (mpiaijcusparse.cu:108)
==27786==    by 0xA20CC3: MatMult (matrix.c:2242)
==27786==    by 0x4645E4: main (ex7.c:93)
==27786==  Address 0x28634560 is 1,616 bytes inside a block of size 1,620 alloc'd
==27786==    at 0x4C26548: memalign (vg_replace_malloc.c:727)
==27786==    by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*, char const*, void**) (mal.c:27)
==27786==    by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char const*, char const*, void**) (mtr.c:186)
==27786==    by 0x5A5296: VecScatterCreate (vscat.c:1168)
==27786==    by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
==27786==    by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType) (mpiaij.c:706)
==27786==    by 0xA45358: MatAssemblyEnd (matrix.c:4959)
==27786==    by 0x464301: main (ex7.c:78)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF97DB: PetscAbsScalar(double) (petscmath.h:206)
==27786==    by 0xCF9878: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x542479: VecValidValues (rvector.c:32)
==27786==    by 0x1105581: PCApply (precon.c:434)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF9880: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x542479: VecValidValues (rvector.c:32)
==27786==    by 0x1105581: PCApply (precon.c:434)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC4E8DF: KSPSolve_GMRES(_p_KSP*) (gmres.c:234)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF97DB: PetscAbsScalar(double) (petscmath.h:206)
==27786==    by 0xCF988B: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x542479: VecValidValues (rvector.c:32)
==27786==    by 0x1105581: PCApply (precon.c:434)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF9893: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x542479: VecValidValues (rvector.c:32)
==27786==    by 0x1105581: PCApply (precon.c:434)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC4E8DF: KSPSolve_GMRES(_p_KSP*) (gmres.c:234)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF97DB: PetscAbsScalar(double) (petscmath.h:206)
==27786==    by 0xCF9878: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x5424F1: VecValidValues (rvector.c:34)
==27786==    by 0x1105972: PCApply (precon.c:442)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF9880: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x5424F1: VecValidValues (rvector.c:34)
==27786==    by 0x1105972: PCApply (precon.c:442)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC4E8DF: KSPSolve_GMRES(_p_KSP*) (gmres.c:234)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF97DB: PetscAbsScalar(double) (petscmath.h:206)
==27786==    by 0xCF988B: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x5424F1: VecValidValues (rvector.c:34)
==27786==    by 0x1105972: PCApply (precon.c:442)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786== 
==27786== Conditional jump or move depends on uninitialised value(s)
==27786==    at 0xCF9893: PetscIsInfOrNanScalar (mathinf.c:67)
==27786==    by 0x5424F1: VecValidValues (rvector.c:34)
==27786==    by 0x1105972: PCApply (precon.c:442)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC33E36: KSPSolve_BCGS (bcgs.c:50)
==27786==    by 0xBAB71C: KSPSolve (itfunc.c:432)
==27786==    by 0xA9C0EF: PCApply_BJacobi_Multiblock(_p_PC*, _p_Vec*, _p_Vec*) (bjacobi.c:945)
==27786==    by 0x11057E6: PCApply (precon.c:440)
==27786==    by 0x1131693: KSP_PCApply(_p_KSP*, _p_Vec*, _p_Vec*) (kspimpl.h:227)
==27786==    by 0x1132479: KSPInitialResidual (itres.c:64)
==27786==    by 0xC4E8DF: KSPSolve_GMRES(_p_KSP*) (gmres.c:234)
==27786== 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: CUSP error 61!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT Date: 2014-01-20 14:12:11 -0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./ex7 on a pargpudbg named ivy.txcorp.com by dmeiser Wed Jan 22 10:23:36 2014
[0]PETSC ERROR: Libraries linked from /scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg/lib
[0]PETSC ERROR: Configure run at Tue Jan 21 16:53:42 2014
[0]PETSC ERROR: Configure options --with-cmake=/scr_ivy/dmeiser/PTSOLVE/cmake/bin/cmake --prefix=/scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg --with-precision=double --with-scalar-type=real --with-fortran-kernels=1 --with-x=no --with-mpi=yes --with-mpi-dir=/scr_ivy/dmeiser/PTSOLVE/openmpi/ --with-openmp=yes --with-valgrind=1 --with-shared-libraries=0 --with-c-support=yes --with-debugging=yes --with-cuda=1 --with-cuda-dir=/usr/local/cuda --with-cuda-arch=sm_35 --download-txpetscgpu --with-thrust=yes --with-thrust-dir=/usr/local/cuda/include --with-umfpack=yes --download-umfpack --with-mumps=yes --with-superlu=yes --download-superlu=yes --download-mumps=yes --download-scalapack --download-parmetis --download-metis --with-cusp=yes --with-cusp-dir=/scr_ivy/dmeiser/PTSOLVE/cusp/include --CUDAFLAGS="-O3 -I/usr/local/cuda/include   --generate-code arch=compute_20,code=sm_20   --generate-code arch=compute_20,code=sm_21   --generate-code arch=compute_30,code=sm_30   --generate-code arch=compute_35,code=sm_35" --with-clanguage=C++ --CFLAGS="-pipe -fPIC" --CXXFLAGS="-pipe -fPIC" --with-c2html=0 --with-gelus=1 --with-gelus-dir=/scr_ivy/dmeiser/software/gelus
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: VecCUSPAllocateCheck() line 72 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecCUSPCopyToGPU() line 96 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecCUSPGetArrayReadWrite() line 1946 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecAXPBYPCZ_SeqCUSP() line 1507 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecAXPBYPCZ() line 726 in /scr_ivy/dmeiser/petsc/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: KSPSolve_BCGS() line 120 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: PCApply_BJacobi_Multiblock() line 945 in /scr_ivy/dmeiser/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: PCApply() line 440 in /scr_ivy/dmeiser/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApply() line 227 in /scr_ivy/dmeiser/petsc/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPInitialResidual() line 64 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 234 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: main() line 209 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/examples/tutorials/ex7.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 76.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
==27786== 
==27786== HEAP SUMMARY:
==27786==     in use at exit: 172,193,472 bytes in 211,364 blocks
==27786==   total heap usage: 350,508 allocs, 139,144 frees, 200,031,576 bytes allocated
==27786== 
==27786== LEAK SUMMARY:
==27786==    definitely lost: 954 bytes in 28 blocks
==27786==    indirectly lost: 61 bytes in 7 blocks
==27786==      possibly lost: 2,154,848 bytes in 15,843 blocks
==27786==    still reachable: 170,037,609 bytes in 195,486 blocks
==27786==         suppressed: 0 bytes in 0 blocks
==27786== Rerun with --leak-check=full to see details of leaked memory
==27786== 
==27786== For counts of detected and suppressed errors, rerun with: -v
==27786== Use --track-origins=yes to see where uninitialised values come from
==27786== ERROR SUMMARY: 82 errors from 10 contexts (suppressed: 6 from 6)

From jed at jedbrown.org  Wed Jan 22 11:31:34 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 10:31:34 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <m2d2jkaq3x.fsf@gmail.com>
References: <87zjmu1ove.fsf@jedbrown.org> <871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
Message-ID: <8738kfexe1.fsf@jedbrown.org>

Sean Farley <sean.michael.farley at gmail.com> writes:

> jed at jedbrown.org writes:
>
>> Sean Farley <sean.michael.farley at gmail.com> writes:
>>>>   $prefix/bin/petsc-${version}-${arch}-config
>>>>
>>>> and symlink $prefix/bin/petsc-config to it by default?
>>>>
>>>> Then always namespace libraries and includes?
>>>
>>> This is what I had in mind as well. Though, distributions would remove
>>> the symlink since that would conflict.
>>
>> Debian would probably manage it through /etc/alternatives.
>
> Just a note that MacPorts would manage it through etc/select :-)

Thanks, Sean.  (I really was asking you as much as Barry/Matt/Satish/etc.
Packaging is a different environment.)

What do the rest of you think?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/8a6bfef2/attachment.sig>

From knepley at gmail.com  Wed Jan 22 11:32:30 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 22 Jan 2014 11:32:30 -0600
Subject: [petsc-dev] How do I build a shared library for exodus?
In-Reply-To: <alpine.LFD.2.11.1401221123110.2367@asterix>
References: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
	<87lhy8haob.fsf@jedbrown.org>
	<28EB76F8-25FB-45D4-B482-543101E7A0D8@lsu.edu>
	<alpine.LFD.2.11.1401221123110.2367@asterix>
Message-ID: <CAMYG4G=Eb3TvoN57c8wf2PU6HjW-L4C8dLUvSmC0T1QvGXWc_A@mail.gmail.com>

On Wed, Jan 22, 2014 at 11:28 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Wed, 22 Jan 2014, Blaise A Bourdin wrote:
>
> >
> > On Jan 21, 2014, at 11:01 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> > > Blaise A Bourdin <bourdin at lsu.edu> writes:
> > >
> > >> Hi,
> > >>
> > >> For some reason, I would need to generate a shared library for
> > >> exodusII, but the Makefile included in the source distribution does
> > >> not have a rule for this. Is there an easy way to modify the build
> > >> system package in order to generate the shared library?
> > >
> > > The Makefile.standalone does not have rules for shared.  Perhaps
> > > upstream exodus (now at version 6.02) is recommending the CMake build?
>
> If upstream recommends cmake - perhaps we should stick to that? We
> already have a bunch of packages that require cmake.
>

I of course disagree. We can use the PETSc shared build rules.

   Matt


> balay at asterix /home/balay/petsc (master)
> $ git grep framework.require |grep PETSc.packages.cmake
> config/PETSc/packages/clique.py:    self.cmake           =
> framework.require('PETSc.packages.cmake',self)
> config/PETSc/packages/elemental.py:    self.cmake           =
> framework.require('PETSc.packages.cmake',self)
> config/PETSc/packages/metis.py:    self.cmake           =
> framework.require('PETSc.packages.cmake',self)
> config/PETSc/packages/parmetis.py:    self.cmake           =
> framework.require('PETSc.packages.cmake',self)
> config/cmakeboot.py:   self.cmake         =
> self.framework.require('PETSc.packages.cmake',       None)
>
>
> > > I'd rather not have exodusii.py depend on CMake if we can help it
> > > (buggy, bad logging, another thing to compile), so maybe we need to add
> > > a shared rule?  Or embrace the beast?
> >
> > let?s say we do want to mess with cmake. What is the typical way to add
> the shared rule?
> >    - patch Makefile.standalone in exodusii.py?
> >    - repackage exodus and serve it from the petsc servers?
>
> Perhaps upstream exodus tarball should support sharedlibrary/static
> library as a build option? elemental.py appears to have the followig
> code commented..
>
> if self.sharedLibraries.useShared:
>   args.append('-DSHARE_LIBRARIES=ON')
>
> Satish
>
> > Is there a BuildSystem way to know that the linker flag to build a
> shared library is -shared under linux and -dynamiclib under mac OS? If so,
> can we simply build the shared library in exodusii.py?
> >
> > Blaise
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/0de07645/attachment.html>

From jed at jedbrown.org  Wed Jan 22 11:35:48 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 10:35:48 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52DFFCE4.1020503@usgs.gov>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org> <52DFFCE4.1020503@usgs.gov>
Message-ID: <87wqhrdimj.fsf@jedbrown.org>

Brad Aagaard <baagaard at usgs.gov> writes:
> We have plans to add higher order basis functions to PyLith this spring, 
> and recognize the deficiencies in many of the available formats. It 
> would be nice to have a consistent way to deal with DMPlex I/O and these 
> other issues (boundaries, high order, etc) across various codes using 
> DMPlex.

How important is it to be able to visualize "checkpoint" files, or to
restart from "visualization" files?  (Many applications write totally
different files because there is no way to include the semantic
information in vis files.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/c58efbbc/attachment.sig>

From paulmullowney at gmail.com  Wed Jan 22 11:37:54 2014
From: paulmullowney at gmail.com (Paul Mullowney)
Date: Wed, 22 Jan 2014 10:37:54 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <52E00098.4050601@txcorp.com>
References: <52DFF67C.5000504@txcorp.com>
	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
	<52E00098.4050601@txcorp.com>
Message-ID: <CAMJ8fwotWKhifhbwB23Lx2Qg-GXLeaZn1e2ozE48mcooAb-MUg@mail.gmail.com>

Hmmm. I may not have protected against the case where the mpaijcusp(arse)
classes are called but without mpirun/mpiexec. I suppose it should have
occurred to me that someone would do this.

try :
mpirun -n 1 ./ex7 -mat_type mpiaijcusparse -vec_type cusp

In this scenario, the sequential to sequential vecscatters should be called.

Then,
mpirun -n 2 ../ex7 -mat_type mpiaijcusparse -vec_type cusp

In this scenario, MPI_General vecscatters should be called ... and work
correctly if you have a system with multiple GPUs.

I

-Paul


On Wed, Jan 22, 2014 at 10:32 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:

> Hey Paul,
>
> Thanks for providing background on this.
>
>
> On Wed 22 Jan 2014 10:05:13 AM MST, Paul Mullowney wrote:
>
>>
>> Dominic,
>> A few years ago, I was trying to minimize the amount of data transfer
>> to and from the GPU (for multi-GPU MatMult) by inspecting the indices
>> of the data that needed to be message to and from the device. Then, I
>> would call gather kernels on the GPU which pulled the scattered data
>> into contiguous buffers and then be transferred to the host
>> asynchronously (while the MatMult was occurring). The existence of
>> VecScatterInitializeForGPU was added in order to build the necessary
>> buffers as needed. This was the motivation behind the existence of
>> VecScatterInitializeForGPU.
>> An alternative approach is to message the smallest contiguous buffer
>> containing all the data with a single cudaMemcpyAsync. This is the
>> method currently implemented.
>> I never found a case where the former implementation (with a GPU
>> gather-kernel) performed better than the alternative approach which
>> messaged the smallest contiguous buffer. I looked at many, many matrices.
>> Now, as far as I understand the VecScatter kernels, this method should
>> only get called if the transfer is MPI_General (i.e. PtoP parallel to
>> parallel). Other VecScatter methods are called in other circumstances
>> where the the scatter is not MPI_General. That assumption could be
>> wrong though.
>>
>
>
> I see. I figured there was some logic in place to make sure that this
> function only gets called in cases where the transfer type is MPI_General.
> I'm getting segfaults in this function where the todata and fromdata are of
> a different type. This could easily be user error but I'm not sure. Here is
> an example valgrind error:
>
> ==27781== Invalid read of size 8
> ==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
> ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*) (
> mpiaijcusparse.cu:108)
> ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
> ==27781== by 0x4645E4: main (ex7.c:93)
> ==27781== Address 0x286305e0 is 1,616 bytes inside a block of size 1,620
> alloc'd
> ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
> ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*,
> char const*, void**) (mal.c:27)
> ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char
> const*, char const*, void**) (mtr.c:186)
> ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
> ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
> ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType)
> (mpiaij.c:706)
> ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
> ==27781== by 0x464301: main (ex7.c:78)
>
> This was produced by src/ksp/ksp/tutorials/ex7.c. The command line options
> are
>
> ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>
> In this particular case the todata is of type VecScatter_Seq_Stride and
> fromdata is of type VecScatter_Seq_General. The complete valgrind log
> (including configure options for petsc) is attached.
>
> Any comments or suggestions are appreciated.
> Cheers,
> Dominic
>
>
>> -Paul
>>
>>
>> On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser <dmeiser at txcorp.com
>> <mailto:dmeiser at txcorp.com>> wrote:
>>
>> Hi,
>>
>> I'm trying to understand VecScatterInitializeForGPU in
>> src/vec/vec/utils/veccusp/__vscatcusp.c. I don't understand why
>>
>> this function can get away with casting the fromdata and todata in
>> the inctx to VecScatter_MPI_General. Don't we need to inspect the
>> VecScatterType fields of the todata and fromdata?
>>
>> Cheers,
>> Dominic
>>
>> --
>> Dominic Meiser
>> Tech-X Corporation
>> 5621 Arapahoe Avenue
>> Boulder, CO 80303
>> USA
>> Telephone: 303-996-2036 <tel:303-996-2036>
>> Fax: 303-448-7756 <tel:303-448-7756>
>> www.txcorp.com <http://www.txcorp.com>
>>
>>
>>
>
>
> --
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/41553408/attachment.html>

From jed at jedbrown.org  Wed Jan 22 11:37:36 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 10:37:36 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org>
	<6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9@lsu.edu>
Message-ID: <87txcvdijj.fsf@jedbrown.org>

Blaise A Bourdin <bourdin at lsu.edu> writes:
> Does anybody knows of any open and widespread format supporting
> elements of order >2? I checked quickly, and according to their
> documentation it seems that none of exodusii, silo 4.7, or xdmf
> support them...

No, I looked hard and everyone I know rolled their own, so that's what I
did (I only needed hexes).

How important is it to support hanging nodes?  Is 2-1 refinement
sufficient?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/b9340bbc/attachment.sig>

From jed at jedbrown.org  Wed Jan 22 11:41:17 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 10:41:17 -0700
Subject: [petsc-dev] How do I build a shared library for exodus?
In-Reply-To: <alpine.LFD.2.11.1401221123110.2367@asterix>
References: <DDA2B9DF-2FD9-4094-9A44-38E5D6EDFF1E@lsu.edu>
	<87lhy8haob.fsf@jedbrown.org>
	<28EB76F8-25FB-45D4-B482-543101E7A0D8@lsu.edu>
	<alpine.LFD.2.11.1401221123110.2367@asterix>
Message-ID: <87r47zdide.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> If upstream recommends cmake - perhaps we should stick to that? We
> already have a bunch of packages that require cmake.

I find it an annoyance, but working around upstream-tested build process
is not a good use of time.

> Perhaps upstream exodus tarball should support sharedlibrary/static
> library as a build option? elemental.py appears to have the followig
> code commented..
>
> if self.sharedLibraries.useShared:
>   args.append('-DSHARE_LIBRARIES=ON')          

This is a typo, should be -DSHARED_LIBRARIES=ON.

For exodus, it's the cmake-standard -DBUILD_SHARED_LIBS=ON.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/484a9558/attachment.sig>

From dmeiser at txcorp.com  Wed Jan 22 11:48:42 2014
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Wed, 22 Jan 2014 10:48:42 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <CAMJ8fwotWKhifhbwB23Lx2Qg-GXLeaZn1e2ozE48mcooAb-MUg@mail.gmail.com>
References: <52DFF67C.5000504@txcorp.com>
	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
	<52E00098.4050601@txcorp.com>
	<CAMJ8fwotWKhifhbwB23Lx2Qg-GXLeaZn1e2ozE48mcooAb-MUg@mail.gmail.com>
Message-ID: <52E0047A.2010404@txcorp.com>

Attached are the logs with 1 rank and 2 ranks. As far as I can tell 
these are different errors.

For the log attached to the previous email I chose to run ex7 without 
mpirun so that valgrind checks ex7 and not mpirun. Is there a way to 
have valgrind check the mpi processes rather than mpirun?

Cheers,
Dominic


On 01/22/2014 10:37 AM, Paul Mullowney wrote:
> Hmmm. I may not have protected against the case where the 
> mpaijcusp(arse) classes are called but without mpirun/mpiexec. I 
> suppose it should have occurred to me that someone would do this.
> try :
> mpirun -n 1 ./ex7 -mat_type mpiaijcusparse -vec_type cusp
> In this scenario, the sequential to sequential vecscatters should be 
> called.
> Then,
> mpirun -n 2 ../ex7 -mat_type mpiaijcusparse -vec_type cusp
> In this scenario, MPI_General vecscatters should be called ... and 
> work correctly if you have a system with multiple GPUs.
> I
> -Paul
>
>
> On Wed, Jan 22, 2014 at 10:32 AM, Dominic Meiser <dmeiser at txcorp.com 
> <mailto:dmeiser at txcorp.com>> wrote:
>
>     Hey Paul,
>
>     Thanks for providing background on this.
>
>
>     On Wed 22 Jan 2014 10:05:13 AM MST, Paul Mullowney wrote:
>
>
>         Dominic,
>         A few years ago, I was trying to minimize the amount of data
>         transfer
>         to and from the GPU (for multi-GPU MatMult) by inspecting the
>         indices
>         of the data that needed to be message to and from the device.
>         Then, I
>         would call gather kernels on the GPU which pulled the
>         scattered data
>         into contiguous buffers and then be transferred to the host
>         asynchronously (while the MatMult was occurring). The existence of
>         VecScatterInitializeForGPU was added in order to build the
>         necessary
>         buffers as needed. This was the motivation behind the existence of
>         VecScatterInitializeForGPU.
>         An alternative approach is to message the smallest contiguous
>         buffer
>         containing all the data with a single cudaMemcpyAsync. This is the
>         method currently implemented.
>         I never found a case where the former implementation (with a GPU
>         gather-kernel) performed better than the alternative approach
>         which
>         messaged the smallest contiguous buffer. I looked at many,
>         many matrices.
>         Now, as far as I understand the VecScatter kernels, this
>         method should
>         only get called if the transfer is MPI_General (i.e. PtoP
>         parallel to
>         parallel). Other VecScatter methods are called in other
>         circumstances
>         where the the scatter is not MPI_General. That assumption could be
>         wrong though.
>
>
>
>     I see. I figured there was some logic in place to make sure that
>     this function only gets called in cases where the transfer type is
>     MPI_General. I'm getting segfaults in this function where the
>     todata and fromdata are of a different type. This could easily be
>     user error but I'm not sure. Here is an example valgrind error:
>
>     ==27781== Invalid read of size 8
>     ==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
>     ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*,
>     _p_Vec*) (mpiaijcusparse.cu:108 <http://mpiaijcusparse.cu:108>)
>     ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
>     ==27781== by 0x4645E4: main (ex7.c:93)
>     ==27781== Address 0x286305e0 is 1,616 bytes inside a block of size
>     1,620 alloc'd
>     ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
>     ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char
>     const*, char const*, void**) (mal.c:27)
>     ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int,
>     char const*, char const*, void**) (mtr.c:186)
>     ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
>     ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
>     ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*,
>     MatAssemblyType) (mpiaij.c:706)
>     ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
>     ==27781== by 0x464301: main (ex7.c:78)
>
>     This was produced by src/ksp/ksp/tutorials/ex7.c. The command line
>     options are
>
>     ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>
>     In this particular case the todata is of type
>     VecScatter_Seq_Stride and fromdata is of type
>     VecScatter_Seq_General. The complete valgrind log (including
>     configure options for petsc) is attached.
>
>     Any comments or suggestions are appreciated.
>     Cheers,
>     Dominic
>
>
>         -Paul
>
>
>         On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser
>         <dmeiser at txcorp.com <mailto:dmeiser at txcorp.com>
>         <mailto:dmeiser at txcorp.com <mailto:dmeiser at txcorp.com>>> wrote:
>
>         Hi,
>
>         I'm trying to understand VecScatterInitializeForGPU in
>         src/vec/vec/utils/veccusp/__vscatcusp.c. I don't understand why
>
>         this function can get away with casting the fromdata and todata in
>         the inctx to VecScatter_MPI_General. Don't we need to inspect the
>         VecScatterType fields of the todata and fromdata?
>
>         Cheers,
>         Dominic
>
>         -- 
>         Dominic Meiser
>         Tech-X Corporation
>         5621 Arapahoe Avenue
>         Boulder, CO 80303
>         USA
>         Telephone: 303-996-2036 <tel:303-996-2036> <tel:303-996-2036
>         <tel:303-996-2036>>
>         Fax: 303-448-7756 <tel:303-448-7756> <tel:303-448-7756
>         <tel:303-448-7756>>
>         www.txcorp.com <http://www.txcorp.com> <http://www.txcorp.com>
>
>
>
>
>
>     -- 
>     Dominic Meiser
>     Tech-X Corporation
>     5621 Arapahoe Avenue
>     Boulder, CO 80303
>     USA
>     Telephone: 303-996-2036 <tel:303-996-2036>
>     Fax: 303-448-7756 <tel:303-448-7756>
>     www.txcorp.com <http://www.txcorp.com>
>
>


-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/388f3a16/attachment.html>
-------------- next part --------------
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Trying to zero at a null pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT Date: 2014-01-20 14:12:11 -0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./ex7 on a pargpudbg named ivy.txcorp.com by dmeiser Wed Jan 22 10:47:22 2014
[0]PETSC ERROR: Libraries linked from /scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg/lib
[0]PETSC ERROR: Configure run at Tue Jan 21 16:53:42 2014
[0]PETSC ERROR: Configure options --with-cmake=/scr_ivy/dmeiser/PTSOLVE/cmake/bin/cmake --prefix=/scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg --with-precision=double --with-scalar-type=real --with-fortran-kernels=1 --with-x=no --with-mpi=yes --with-mpi-dir=/scr_ivy/dmeiser/PTSOLVE/openmpi/ --with-openmp=yes --with-valgrind=1 --with-shared-libraries=0 --with-c-support=yes --with-debugging=yes --with-cuda=1 --with-cuda-dir=/usr/local/cuda --with-cuda-arch=sm_35 --download-txpetscgpu --with-thrust=yes --with-thrust-dir=/usr/local/cuda/include --with-umfpack=yes --download-umfpack --with-mumps=yes --with-superlu=yes --download-superlu=yes --download-mumps=yes --download-scalapack --download-parmetis --download-metis --with-cusp=yes --with-cusp-dir=/scr_ivy/dmeiser/PTSOLVE/cusp/include --CUDAFLAGS="-O3 -I/usr/local/cuda/include   --generate-code arch=compute_20,code=sm_20   --generate-code arch=compute_20,code=sm_21   --generate-code arch=compute_30,code=sm_30   --generate-code arch=compute_35,code=sm_35" --with-clanguage=C++ --CFLAGS="-pipe -fPIC" --CXXFLAGS="-pipe -fPIC" --with-c2html=0 --with-gelus=1 --with-gelus-dir=/scr_ivy/dmeiser/software/gelus
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PetscMemzero() line 1930 in /scr_ivy/dmeiser/petsc/include/petscsys.h
[0]PETSC ERROR: VecSet_Seq() line 729 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/dvec2.c
[0]PETSC ERROR: VecSet() line 575 in /scr_ivy/dmeiser/petsc/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: KSPSolve() line 417 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: PCApply_BJacobi_Multiblock() line 945 in /scr_ivy/dmeiser/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: PCApply() line 440 in /scr_ivy/dmeiser/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApply() line 227 in /scr_ivy/dmeiser/petsc/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPInitialResidual() line 64 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 234 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Null argument, when expecting valid pointer!
[1]PETSC ERROR: Trying to zero at a null pointer!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT Date: 2014-01-20 14:12:11 -0700
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ./ex7 on a pargpudbg named ivy.txcorp.com by dmeiser Wed Jan 22 10:47:22 2014
[1]PETSC ERROR: Libraries linked from /scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg/lib
[1]PETSC ERROR: Configure run at Tue Jan 21 16:53:42 2014
[1]PETSC ERROR: Configure options --with-cmake=/scr_ivy/dmeiser/PTSOLVE/cmake/bin/cmake --prefix=/scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg --with-precision=double --with-scalar-type=real --with-fortran-kernels=1 --with-x=no --with-mpi=yes --with-mpi-dir=/scr_ivy/dmeiser/PTSOLVE/openmpi/ --with-openmp=yes --with-valgrind=1 --with-shared-libraries=0 --with-c-support=yes --with-debugging=yes --with-cuda=1 --with-cuda-dir=/usr/local/cuda --with-cuda-arch=sm_35 --download-txpetscgpu --with-thrust=yes --with-thrust-dir=/usr/local/cuda/include --with-umfpack=yes --download-umfpack --with-mumps=yes --with-superlu=yes --download-superlu=yes --download-mumps=yes --download-scalapack --download-parmetis --download-metis --with-cusp=yes --with-cusp-dir=/scr_ivy/dmeiser/PTSOLVE/cusp/include --CUDAFLAGS="-O3 -I/usr/local/cuda/include   --generate-code arch=compute_20,code=sm_20   --generate-code arch=compute_20,code=sm_21   --generate-code arch=compute_30,code=sm_30   --generate-code arch=compute_35,code=sm_35" --with-clanguage=C++ --CFLAGS="-pipe -fPIC" --CXXFLAGS="-pipe -fPIC" --with-c2html=0 --with-gelus=1 --with-gelus-dir=/scr_ivy/dmeiser/software/gelus
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: PetscMemzero() line 1930 in /scr_ivy/dmeiser/petsc/include/petscsys.h
[1]PETSC ERROR: VecSet_Seq() line 729 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/dvec2.c
[1]PETSC ERROR: VecSet() line 575 in /scr_ivy/dmeiser/petsc/src/vec/vec/interface/rvector.c
[1]PETSC ERROR: KSPSolve() line 417 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: PCApply_BJacobi_Multiblock() line 945 in /scr_ivy/dmeiser/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[1]PETSC ERROR: PCApply() line 440 in /scr_ivy/dmeiser/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: main() line 209 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/examples/tutorials/ex7.c
[1]PETSC ERROR: KSP_PCApply() line 227 in /scr_ivy/dmeiser/petsc/include/petsc-private/kspimpl.h
[1]PETSC ERROR: KSPInitialResidual() line 64 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itres.c
[1]PETSC ERROR: KSPSolve_GMRES() line 234 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: main() line 209 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/examples/tutorials/ex7.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD 
with errorcode 85.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 27839 on
node ivy.txcorp.com exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[ivy.txcorp.com:27838] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[ivy.txcorp.com:27838] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
-------------- next part --------------
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: CUSP error 61!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT Date: 2014-01-20 14:12:11 -0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./ex7 on a pargpudbg named ivy.txcorp.com by dmeiser Wed Jan 22 10:47:15 2014
[0]PETSC ERROR: Libraries linked from /scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg/lib
[0]PETSC ERROR: Configure run at Tue Jan 21 16:53:42 2014
[0]PETSC ERROR: Configure options --with-cmake=/scr_ivy/dmeiser/PTSOLVE/cmake/bin/cmake --prefix=/scr_ivy/dmeiser/petsc-gpu-dev/build/pargpudbg --with-precision=double --with-scalar-type=real --with-fortran-kernels=1 --with-x=no --with-mpi=yes --with-mpi-dir=/scr_ivy/dmeiser/PTSOLVE/openmpi/ --with-openmp=yes --with-valgrind=1 --with-shared-libraries=0 --with-c-support=yes --with-debugging=yes --with-cuda=1 --with-cuda-dir=/usr/local/cuda --with-cuda-arch=sm_35 --download-txpetscgpu --with-thrust=yes --with-thrust-dir=/usr/local/cuda/include --with-umfpack=yes --download-umfpack --with-mumps=yes --with-superlu=yes --download-superlu=yes --download-mumps=yes --download-scalapack --download-parmetis --download-metis --with-cusp=yes --with-cusp-dir=/scr_ivy/dmeiser/PTSOLVE/cusp/include --CUDAFLAGS="-O3 -I/usr/local/cuda/include   --generate-code arch=compute_20,code=sm_20   --generate-code arch=compute_20,code=sm_21   --generate-code arch=compute_30,code=sm_30   --generate-code arch=compute_35,code=sm_35" --with-clanguage=C++ --CFLAGS="-pipe -fPIC" --CXXFLAGS="-pipe -fPIC" --with-c2html=0 --with-gelus=1 --with-gelus-dir=/scr_ivy/dmeiser/software/gelus
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: VecCUSPAllocateCheck() line 72 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecCUSPCopyToGPU() line 96 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecCUSPGetArrayReadWrite() line 1946 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecAXPBYPCZ_SeqCUSP() line 1507 in /scr_ivy/dmeiser/petsc/src/vec/vec/impls/seq/seqcusp/veccusp.cu
[0]PETSC ERROR: VecAXPBYPCZ() line 726 in /scr_ivy/dmeiser/petsc/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: KSPSolve_BCGS() line 120 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: PCApply_BJacobi_Multiblock() line 945 in /scr_ivy/dmeiser/petsc/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: PCApply() line 440 in /scr_ivy/dmeiser/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApply() line 227 in /scr_ivy/dmeiser/petsc/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPInitialResidual() line 64 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 234 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 432 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: main() line 209 in /scr_ivy/dmeiser/petsc/src/ksp/ksp/examples/tutorials/ex7.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 76.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 27836 on
node ivy.txcorp.com exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

From bourdin at lsu.edu  Wed Jan 22 11:50:44 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Wed, 22 Jan 2014 17:50:44 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <87txcvdijj.fsf@jedbrown.org>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org>
	<6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9@lsu.edu>
	<87txcvdijj.fsf@jedbrown.org>
Message-ID: <544CA09E-D71C-45DD-A5F9-04FD02A133CB@lsu.edu>

On Jan 22, 2014, at 11:37 AM, Jed Brown <jed at jedbrown.org> wrote:

> Blaise A Bourdin <bourdin at lsu.edu> writes:
>> Does anybody knows of any open and widespread format supporting
>> elements of order >2? I checked quickly, and according to their
>> documentation it seems that none of exodusii, silo 4.7, or xdmf
>> support them...
> 
> No, I looked hard and everyone I know rolled their own, so that's what I
> did (I only needed hexes).
Did you write a whole new format? How about simply extendinf xdmf to add higher order element and send patches supporting these extensions to visit / paraview?

> How important is it to support hanging nodes?  Is 2-1 refinement
> sufficient?
Can hanging nodes be handled through cell sets of various element types?

Blaise 

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









From paulmullowney at gmail.com  Wed Jan 22 11:54:28 2014
From: paulmullowney at gmail.com (Paul Mullowney)
Date: Wed, 22 Jan 2014 10:54:28 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <52E0047A.2010404@txcorp.com>
References: <52DFF67C.5000504@txcorp.com>
	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
	<52E00098.4050601@txcorp.com>
	<CAMJ8fwotWKhifhbwB23Lx2Qg-GXLeaZn1e2ozE48mcooAb-MUg@mail.gmail.com>
	<52E0047A.2010404@txcorp.com>
Message-ID: <CAMJ8fwqUmOSisnhkiGdkdFTMOvEw_ma1JBwrqfHBTnFQ=EkMnw@mail.gmail.com>

Oh. You're opening a can of worms but maybe that's your intent ;) I see the
block Jacobi preconditioner in the valgrind logs.

Do,
mpirun -n 1 (or 2) ./ex7 -mat_type mpiaijcusparse -vec_type mpicusp
-pc_type none

>From here, we can try to sort out the VecScatterInitializeForGPU problem
when mpirun/exec is not used.

If you want to implement block jacobi preconditioner on multiple GPUs,
that's a larger problem to solve. I had some code that sort of worked. We'd
have to sit down and discuss.
-Paul


On Wed, Jan 22, 2014 at 10:48 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:

>  Attached are the logs with 1 rank and 2 ranks. As far as I can tell
> these are different errors.
>
> For the log attached to the previous email I chose to run ex7 without
> mpirun so that valgrind checks ex7 and not mpirun. Is there a way to have
> valgrind check the mpi processes rather than mpirun?
>
> Cheers,
> Dominic
>
>
>
> On 01/22/2014 10:37 AM, Paul Mullowney wrote:
>
>  Hmmm. I may not have protected against the case where the
> mpaijcusp(arse) classes are called but without mpirun/mpiexec. I suppose it
> should have occurred to me that someone would do this.
>
> try :
> mpirun -n 1 ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>
> In this scenario, the sequential to sequential vecscatters should be
> called.
>
> Then,
> mpirun -n 2 ../ex7 -mat_type mpiaijcusparse -vec_type cusp
>
> In this scenario, MPI_General vecscatters should be called ... and work
> correctly if you have a system with multiple GPUs.
>
> I
>
> -Paul
>
>
> On Wed, Jan 22, 2014 at 10:32 AM, Dominic Meiser <dmeiser at txcorp.com>wrote:
>
>> Hey Paul,
>>
>> Thanks for providing background on this.
>>
>>
>> On Wed 22 Jan 2014 10:05:13 AM MST, Paul Mullowney wrote:
>>
>>>
>>> Dominic,
>>> A few years ago, I was trying to minimize the amount of data transfer
>>> to and from the GPU (for multi-GPU MatMult) by inspecting the indices
>>> of the data that needed to be message to and from the device. Then, I
>>> would call gather kernels on the GPU which pulled the scattered data
>>> into contiguous buffers and then be transferred to the host
>>> asynchronously (while the MatMult was occurring). The existence of
>>> VecScatterInitializeForGPU was added in order to build the necessary
>>> buffers as needed. This was the motivation behind the existence of
>>> VecScatterInitializeForGPU.
>>> An alternative approach is to message the smallest contiguous buffer
>>> containing all the data with a single cudaMemcpyAsync. This is the
>>> method currently implemented.
>>> I never found a case where the former implementation (with a GPU
>>> gather-kernel) performed better than the alternative approach which
>>> messaged the smallest contiguous buffer. I looked at many, many matrices.
>>> Now, as far as I understand the VecScatter kernels, this method should
>>> only get called if the transfer is MPI_General (i.e. PtoP parallel to
>>> parallel). Other VecScatter methods are called in other circumstances
>>> where the the scatter is not MPI_General. That assumption could be
>>> wrong though.
>>>
>>
>>
>>  I see. I figured there was some logic in place to make sure that this
>> function only gets called in cases where the transfer type is MPI_General.
>> I'm getting segfaults in this function where the todata and fromdata are of
>> a different type. This could easily be user error but I'm not sure. Here is
>> an example valgrind error:
>>
>> ==27781== Invalid read of size 8
>> ==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
>> ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*) (
>> mpiaijcusparse.cu:108)
>> ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
>> ==27781== by 0x4645E4: main (ex7.c:93)
>> ==27781== Address 0x286305e0 is 1,616 bytes inside a block of size 1,620
>> alloc'd
>> ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
>> ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*,
>> char const*, void**) (mal.c:27)
>> ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char
>> const*, char const*, void**) (mtr.c:186)
>> ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
>> ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
>> ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType)
>> (mpiaij.c:706)
>> ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
>> ==27781== by 0x464301: main (ex7.c:78)
>>
>> This was produced by src/ksp/ksp/tutorials/ex7.c. The command line
>> options are
>>
>> ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>>
>> In this particular case the todata is of type VecScatter_Seq_Stride and
>> fromdata is of type VecScatter_Seq_General. The complete valgrind log
>> (including configure options for petsc) is attached.
>>
>> Any comments or suggestions are appreciated.
>> Cheers,
>> Dominic
>>
>>
>>> -Paul
>>>
>>>
>>> On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser <dmeiser at txcorp.com
>>>  <mailto:dmeiser at txcorp.com>> wrote:
>>>
>>> Hi,
>>>
>>> I'm trying to understand VecScatterInitializeForGPU in
>>>  src/vec/vec/utils/veccusp/__vscatcusp.c. I don't understand why
>>>
>>> this function can get away with casting the fromdata and todata in
>>> the inctx to VecScatter_MPI_General. Don't we need to inspect the
>>> VecScatterType fields of the todata and fromdata?
>>>
>>> Cheers,
>>> Dominic
>>>
>>> --
>>> Dominic Meiser
>>> Tech-X Corporation
>>> 5621 Arapahoe Avenue
>>> Boulder, CO 80303
>>> USA
>>>  Telephone: 303-996-2036 <tel:303-996-2036>
>>> Fax: 303-448-7756 <tel:303-448-7756>
>>> www.txcorp.com <http://www.txcorp.com>
>>>
>>>
>>>
>>
>>
>> --
>> Dominic Meiser
>> Tech-X Corporation
>> 5621 Arapahoe Avenue
>> Boulder, CO 80303
>> USA
>> Telephone: 303-996-2036
>> Fax: 303-448-7756
>> www.txcorp.com
>>
>>
>
>
> --
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756www.txcorp.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/7847b18a/attachment.html>

From jed at jedbrown.org  Wed Jan 22 11:57:51 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 10:57:51 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <544CA09E-D71C-45DD-A5F9-04FD02A133CB@lsu.edu>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org>
	<6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9@lsu.edu>
	<87txcvdijj.fsf@jedbrown.org>
	<544CA09E-D71C-45DD-A5F9-04FD02A133CB@lsu.edu>
Message-ID: <87iotbdhls.fsf@jedbrown.org>

Blaise A Bourdin <bourdin at lsu.edu> writes:
> Did you write a whole new format? 

I wrote a new HDF5-based format and a VisIt plugin.

> How about simply extendinf xdmf to add higher order element and send
> patches supporting these extensions to visit / paraview?

This would be a nontrivial extension.  Existing XDMF maps directly to
VTK data structures, but VTK does not natively support high order, so
you have to decompose the high-order elements into low-order
sub-elements.

>> How important is it to support hanging nodes?  Is 2-1 refinement
>> sufficient?
> Can hanging nodes be handled through cell sets of various element types?

How so?  I'm thinking of, say, octree refinement.  How would you
describe that?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/cf85ac37/attachment.sig>

From bourdin at lsu.edu  Wed Jan 22 12:16:57 2014
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Wed, 22 Jan 2014 18:16:57 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52DFFCE4.1020503@usgs.gov>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org> <52DFFCE4.1020503@usgs.gov>
Message-ID: <7C0CA930-7595-4756-B7A3-885ECD6767B3@lsu.edu>




On Jan 22, 2014, at 11:16 AM, Brad Aagaard <baagaard at usgs.gov> wrote:

> On 01/22/2014 08:57 AM, Jed Brown wrote:
>> Moving back to list.
>> 
>> Blaise A Bourdin <bourdin at lsu.edu> writes:
>> 
>>> On Jan 21, 2014, at 11:20 AM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>>   - Reading from or writing to exodus files is not supported.
>>>>>> 
>>>>> 
>>>>> Yes, I think this is the best target. It should be similar to writing HDF5
>>>>> that we do for PyLith.
>>>> 
>>>> How are we going to represent high-order elements in Exodus?  Doesn't it
>>>> only support quadratic continuous spaces?  What about DG spaces, H(div)
>>>> spaces, or higher than second order?
>>> 
>>> It only supports first and second order elements. I don?t think that continuity is an issue. Nothing prevents you to alter the connectivity table.
>>> Element types is set by blocks, so mixing polynomial orders is not possible without some trickery
>> 
>> Do we consider this an acceptable long-term solution?
>> 
> 
> I see a DMPlex layout for HDF-5 files as a convenient means of providing fast, parallel I/O with Xdmf files as a means to provide a way for ParaView/Visit, etc to be able to display much of the info within the HDF5-files without custom plug-ins. Plug-ins might be necessary for specialized information that can't be expressed in Xdmf files. We deal with end-users with limited programming experience for which plug-ins might be difficult.
> 
> For PyLith, I like being able to use h5py for high-level access to the info in the HDF5 files for post processing.
> 
> We have plans to add higher order basis functions to PyLith this spring, and recognize the deficiencies in many of the available formats. It would be nice to have a consistent way to deal with DMPlex I/O and these other issues (boundaries, high order, etc) across various codes using DMPlex.


How about the following strategy:
1. implement Q&D exodus / silo / whatever format based on Jed?s current VTU. It should be a good exercise in understanding PetscSF.
2. write a scalable hdf5 / xdmf scheme starting from the code already in pylith. Last time I checked, the pylith xdmf scheme used multiple files (1 per cell block / time step), and did not allow to preserve continuity of fields across cell sets (because it does not make sense to do so in its applications). Would the scheme from the attached xmf and hdf5 files make sense? It was an old attempt at translating from exodus to xdmf without loss of information.

Blaise
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Beam-0000.xmf
Type: application/octet-stream
Size: 359050 bytes
Desc: Beam-0000.xmf
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/f532c305/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Beam-0000.h5
Type: application/octet-stream
Size: 612320 bytes
Desc: Beam-0000.h5
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/f532c305/attachment-0001.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ATT00001.txt
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/f532c305/attachment.txt>

From tautges at mcs.anl.gov  Wed Jan 22 12:18:54 2014
From: tautges at mcs.anl.gov (Tim Tautges (ANL))
Date: Wed, 22 Jan 2014 12:18:54 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
	<52DEE19A.30306@mcs.anl.gov>
	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
Message-ID: <52E00B8E.2020605@mcs.anl.gov>



On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>
>>>
>>> I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even
>>> when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using
>>> HDF5 as their storage format. Exodus has an parallel extension called nemesis, but I can?t figure out how how
>>> their concept of ghost point and cells works. The documentation on this point is really unclear.
>>>
>
>
> I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel IO front (I?m
> assuming those guys also run large simulations?). That said, I see this as a two stage process: first integrate with
> DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of
> readers/writers. At least this way there will be some agility and interoperability between different formats, and it
> will not be too disruptive to the application codes when a different formats adopted.
>

My impression is that the ExodusII people are working within the context of code frameworks more than disk file formats 
to do this, e.g. in Trilinos and Sierra.  I don't think the ExoII file format by itself is very conducive to 
representing parallel, which is why Nemesis writes an annotation (though, I haven't followed ExoII developments closely 
since they went open source several years back).

>
>
>>> b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la
>>> nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an
>>> un-partitionner as a post-processor.
>
> I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry too much
> about the parallel layout although some clear optimisations are possible. In the  worst case it is a three stage
> process of where we perform a parallel read of the connectivity, scatter/gather for continuous numbering, parallel
> repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.
>

We've seen fragmentation with unstructured meshes being a problem too, and you won't escape that even with renumbering 
(though reading then migrating would address that, at the cost of some additional communication and possibly reading to 
figure out where things need to go).

>>>
>>
>> Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there
>> are other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial)
>> ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been
>> iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process
>> of developing direct support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.
>> MOAB has pretty solid support for things like computing sharing and ghosting between procs and exchanging/reducing
>> field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the
>> mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a plugin you
>> can enable.  See http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff
>> is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>
> Another reason this could be of great interest is that MOAB supports (according to the docs) geometric topology which
> could be used when adapting the mesh on a curved surface for example - another item on my which list.

Yes, MOAB's file format can handle definitions of groupings (and relations between the groups) necessary to represent 
geometric topology.  What you use for the shape evaluation of those geometric surfaces is another question though.  If 
you're wanting to do that using a reconstructed continuous patch, MOAB has some code to do that too, though it's not as 
good as the latest stuff in that area (from Jiao at Stony Brook).


Is it
> integrated to PETSc via the plex or does this essentially replace the functionality of the plex?

It's not via plex, but I'm pretty sure all the mesh-related functionality available through plex is available through 
different API functions in MOAB.

Why does it break
> down for more than 16k procs?

It's a combination of things:
- maximizing generality means we're using more than just 2 or 3 tables, because in addition to nodes and elements, we 
need groupings, whose membership determines whether a group is resident on a given processor, etc, and that strongly 
affects scaling
- that same generality causes us to hit an MPI/IO bug on IBM (though we haven't checked on Q yet to see if that's been 
addressed, it might have been); we've worked with ANL I/O guys off and on on this, and hope to get back to that on Q soon
- we do single file parallel I/O, without any 2-phase (communicate down to I/O nodes then do I/O), and that hits HDF5 
pretty hard; we're working with hdfgroup to explore that

We haven't done any benchmarking on a Lustre system yet, but I expect that to do worse than IBM, because of the many 
tables thing (my impression is that Lustre doesn't handle frequent metadata reads well)

is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM
> codes such as nek500 that can run on ~1m cores?
>

Those codes go for a much more restricted I/O data case, which allows them to specialize and do their own implementation 
of parallel I/O.  So, Nek5000 has its own implementation of poor man's parallel, they repeat vertices in the file that 
are logically the same (shared between hexes), and they don't really do subsets.  I think that's great to do if you have 
to, but I'm still hoping for more support in that direction from general libraries.

>>
>> libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is
>> (but they've been working on it recently I know).
>>
>
>
> Are there any other formats that we should be considering? It?s a few years since I tried playing about with CGNS -
> at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks interesting as it
> is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?
>

Don't know about XDMF.  I know there's been a bunch of work on SILO and its parallel performance fairly recently (3 or 4 
yrs ago) and it's used heavily inside LLNL.

- tim

> Cheers Gerard
>

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706



From knepley at gmail.com  Wed Jan 22 12:45:14 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 22 Jan 2014 12:45:14 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52E00B8E.2020605@mcs.anl.gov>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
	<52DEE19A.30306@mcs.anl.gov>
	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
	<52E00B8E.2020605@mcs.anl.gov>
Message-ID: <CAMYG4GmVRKuy_XryVLcquOAU+0A10pugKWQS2J13rwScb+Mq7w@mail.gmail.com>

Tim,

  I do not consider MOAB a real alternative here.

     Matt

On Wed, Jan 22, 2014 at 12:18 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov>wrote:

>
>
> On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>
>>
>>
>>>> I am not sure if Exodus has a good solution here. As far as I
>>>> understand, exodus is inherently sequential, even
>>>> when implemented with HDF5 instead of netcdf. I would also worry about
>>>> third party support for exodus files using
>>>> HDF5 as their storage format. Exodus has an parallel extension called
>>>> nemesis, but I can?t figure out how how
>>>> their concept of ghost point and cells works. The documentation on this
>>>> point is really unclear.
>>>>
>>>>
>>
>> I have to admit I was kind of hoping that ExodusII folks would have come
>> on a bit more on the parallel IO front (I?m
>> assuming those guys also run large simulations?). That said, I see this
>> as a two stage process: first integrate with
>> DMPlex as that should give the high level abstraction for read/write to
>> file; secondly extend the family of
>> readers/writers. At least this way there will be some agility and
>> interoperability between different formats, and it
>> will not be too disruptive to the application codes when a different
>> formats adopted.
>>
>>
> My impression is that the ExodusII people are working within the context
> of code frameworks more than disk file formats to do this, e.g. in Trilinos
> and Sierra.  I don't think the ExoII file format by itself is very
> conducive to representing parallel, which is why Nemesis writes an
> annotation (though, I haven't followed ExoII developments closely since
> they went open source several years back).
>
>
>
>>
>>  b. Do ?poor man? parallel I/O where each CPU does its own I/O, and
>>>> possibly create interface matching files ? la
>>>> nemesis or SILO. Maybe, we can save enough information on the parallel
>>>> layout in order to easily write an
>>>> un-partitionner as a post-processor.
>>>>
>>>
>> I am pretty sure that if we are writing everything in slabs to a HDF5
>> container we do not have to worry too much
>> about the parallel layout although some clear optimisations are possible.
>> In the  worst case it is a three stage
>> process of where we perform a parallel read of the connectivity,
>> scatter/gather for continuous numbering, parallel
>> repartitioning and subsequent parallel read of remaining data.
>> Importantly, it is at least scalable.
>>
>>
> We've seen fragmentation with unstructured meshes being a problem too, and
> you won't escape that even with renumbering (though reading then migrating
> would address that, at the cost of some additional communication and
> possibly reading to figure out where things need to go).
>
>
>
>>>>
>>> Depending on the degree of direct interaction/automation in those
>>> interactions between the mesh and Petsc, there
>>> are other options as well.  One that we're developing, based on the MOAB
>>> library, can read/write (in serial)
>>> ExodusII, and also supports parallel read/write using its own HDF5-based
>>> format.  Parallel I/O robustness has been
>>> iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller
>>> problems it should work.  We're in the process
>>> of developing direct support for going between a mesh defined with
>>> fields (called tags in MOAB) and petsc vectors.
>>> MOAB has pretty solid support for things like computing sharing and
>>> ghosting between procs and exchanging/reducing
>>> field values on those entities.  Viz is supported either by compiling a
>>> VTK/Paraview plugin that pulls the
>>> mesh/fields through MOAB or by translating to VTK (also supported
>>> directly from MOAB); Visit also has a plugin you
>>> can enable.  See http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for
>>> details of MOAB; the petsc integration stuff
>>> is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>>>
>>
>> Another reason this could be of great interest is that MOAB supports
>> (according to the docs) geometric topology which
>> could be used when adapting the mesh on a curved surface for example -
>> another item on my which list.
>>
>
> Yes, MOAB's file format can handle definitions of groupings (and relations
> between the groups) necessary to represent geometric topology.  What you
> use for the shape evaluation of those geometric surfaces is another
> question though.  If you're wanting to do that using a reconstructed
> continuous patch, MOAB has some code to do that too, though it's not as
> good as the latest stuff in that area (from Jiao at Stony Brook).
>
>
>
> Is it
>
>> integrated to PETSc via the plex or does this essentially replace the
>> functionality of the plex?
>>
>
> It's not via plex, but I'm pretty sure all the mesh-related functionality
> available through plex is available through different API functions in MOAB.
>
>
> Why does it break
>
>> down for more than 16k procs?
>>
>
> It's a combination of things:
> - maximizing generality means we're using more than just 2 or 3 tables,
> because in addition to nodes and elements, we need groupings, whose
> membership determines whether a group is resident on a given processor,
> etc, and that strongly affects scaling
> - that same generality causes us to hit an MPI/IO bug on IBM (though we
> haven't checked on Q yet to see if that's been addressed, it might have
> been); we've worked with ANL I/O guys off and on on this, and hope to get
> back to that on Q soon
> - we do single file parallel I/O, without any 2-phase (communicate down to
> I/O nodes then do I/O), and that hits HDF5 pretty hard; we're working with
> hdfgroup to explore that
>
> We haven't done any benchmarking on a Lustre system yet, but I expect that
> to do worse than IBM, because of the many tables thing (my impression is
> that Lustre doesn't handle frequent metadata reads well)
>
>
> is it just a case that Lustre gets hammered? What magic sauce is used by
> high order FEM
>
>> codes such as nek500 that can run on ~1m cores?
>>
>>
> Those codes go for a much more restricted I/O data case, which allows them
> to specialize and do their own implementation of parallel I/O.  So, Nek5000
> has its own implementation of poor man's parallel, they repeat vertices in
> the file that are logically the same (shared between hexes), and they don't
> really do subsets.  I think that's great to do if you have to, but I'm
> still hoping for more support in that direction from general libraries.
>
>
>
>>> libmesh also maintains its own DMlibmesh, but I'm not sure how solid
>>> their support for large mesh / parallel I/O is
>>> (but they've been working on it recently I know).
>>>
>>>
>>
>> Are there any other formats that we should be considering? It?s a few
>> years since I tried playing about with CGNS -
>> at the time its parallel IO was non-existent and I have not seen it being
>> pushed since. XDMF looks interesting as it
>> is essentially some xml metadata and a HDF5 bucket. Is anyone championing
>> this?
>>
>>
> Don't know about XDMF.  I know there's been a bunch of work on SILO and
> its parallel performance fairly recently (3 or 4 yrs ago) and it's used
> heavily inside LLNL.
>
> - tim
>
>  Cheers Gerard
>>
>>
> --
> ================================================================
> "You will keep in perfect peace him whose mind is
>   steadfast, because he trusts in you."               Isaiah 26:3
>
>              Tim Tautges            Argonne National Laboratory
>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
>             fax: (608) 263-4499      Madison, WI 53706
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/ab7b7474/attachment.html>

From tautges at mcs.anl.gov  Wed Jan 22 12:47:54 2014
From: tautges at mcs.anl.gov (Tim Tautges (ANL))
Date: Wed, 22 Jan 2014 12:47:54 -0600
Subject: [petsc-dev] DMPlex reader / viewers
In-Reply-To: <mailman.11389.1390411789.3278.petsc-dev@mcs.anl.gov>
References: <mailman.11389.1390411789.3278.petsc-dev@mcs.anl.gov>
Message-ID: <52E0125A.1000105@mcs.anl.gov>



On 01/22/2014 11:29 AM, petsc-dev-request at mcs.anl.gov wrote:
> Message: 3 Date: Wed, 22 Jan 2014 17:26:02 +0000 From: Blaise A Bourdin<bourdin at lsu.edu> To: Jed
> Brown<jed at jedbrown.org> Cc: petsc-dev<petsc-dev at mcs.anl.gov> Subject: Re: [petsc-dev] DMplex reader / viewers
> Message-ID:<6A6423C9-5C9A-4C6E-92F8-29122FC2B8F9 at lsu.edu> Content-Type: text/plain; charset="Windows-1252"
>
>
> On Jan 22, 2014, at 10:57 AM, Jed Brown<jed at jedbrown.org>  wrote:
>
>>> Moving back to list.
> sorry, muscle memory prevents me from hitting that reply-to button
>
>>>
>>> Blaise A Bourdin<bourdin at lsu.edu>  writes:
>>>
>>>>> On Jan 21, 2014, at 11:20 AM, Jed Brown<jed at jedbrown.org>  wrote:
>>>>>
>>>>>>> Matthew Knepley<knepley at gmail.com>  writes:
>>>>>>>>> - Reading from or writing to exodus files is not supported.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, I think this is the best target. It should be similar to writing HDF5 that we do for PyLith.
>>>>>>>
>>>>>>> How are we going to represent high-order elements in Exodus?  Doesn't it only support quadratic
>>>>>>> continuous spaces?  What about DG spaces, H(div) spaces, or higher than second order?
>>>>>
>>>>> It only supports first and second order elements. I don?t think that continuity is an issue. Nothing prevents
>>>>> you to alter the connectivity table. Element types is set by blocks, so mixing polynomial orders is not
>>>>> possible without some trickery
>>>
>>> Do we consider this an acceptable long-term solution?
> hell no!
>
> Does anybody knows of any open and widespread format supporting elements of order >2? I checked quickly, and
> according to their documentation it seems that none of exodusii, silo 4.7, or xdmf support them...
>

MOAB supports this, with certain constraints (if one facet of an element is resolved with higher-order nodes, they all 
have to be), including I/O.  But, you'd have to modify any viz tool to handle those elements (e.g. both Paraview and 
Visit interpret Nek5000's Nth order spectral element as NxN linear elements.

- tim

> Incidentally, exodus docs last update is from 2006,silo?s from 2010, andhttp://www.xdmf.org  has seen minor edits
> after 2009... I seem to recall that the project is now hosted with visit. Not sure.
>
> Blaise

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706



From dmeiser at txcorp.com  Wed Jan 22 12:52:43 2014
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Wed, 22 Jan 2014 11:52:43 -0700
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <CAMJ8fwqUmOSisnhkiGdkdFTMOvEw_ma1JBwrqfHBTnFQ=EkMnw@mail.gmail.com>
References: <52DFF67C.5000504@txcorp.com>
	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
	<52E00098.4050601@txcorp.com>
	<CAMJ8fwotWKhifhbwB23Lx2Qg-GXLeaZn1e2ozE48mcooAb-MUg@mail.gmail.com>
	<52E0047A.2010404@txcorp.com>
	<CAMJ8fwqUmOSisnhkiGdkdFTMOvEw_ma1JBwrqfHBTnFQ=EkMnw@mail.gmail.com>
Message-ID: <52E0137B.4050200@txcorp.com>

On Wed 22 Jan 2014 10:54:28 AM MST, Paul Mullowney wrote:
> Oh. You're opening a can of worms but maybe that's your intent ;) I
> see the block Jacobi preconditioner in the valgrind logs.

Didn't mean to open a can of worms.

> Do,
> mpirun -n 1 (or 2) ./ex7 -mat_type mpiaijcusparse -vec_type mpicusp
> -pc_type none

This works.

> From here, we can try to sort out the VecScatterInitializeForGPU
> problem when mpirun/exec is not used.
> If you want to implement block jacobi preconditioner on multiple GPUs,
> that's a larger problem to solve. I had some code that sort of worked.
> We'd have to sit down and discuss.

I'd be really interested in learning more about this.

Cheers,
Dominic


> -Paul
>
>
> On Wed, Jan 22, 2014 at 10:48 AM, Dominic Meiser <dmeiser at txcorp.com
> <mailto:dmeiser at txcorp.com>> wrote:
>
>     Attached are the logs with 1 rank and 2 ranks. As far as I can
>     tell these are different errors.
>
>     For the log attached to the previous email I chose to run ex7
>     without mpirun so that valgrind checks ex7 and not mpirun. Is
>     there a way to have valgrind check the mpi processes rather than
>     mpirun?
>
>     Cheers,
>     Dominic
>
>
>
>     On 01/22/2014 10:37 AM, Paul Mullowney wrote:
>>     Hmmm. I may not have protected against the case where the
>>     mpaijcusp(arse) classes are called but without mpirun/mpiexec. I
>>     suppose it should have occurred to me that someone would do this.
>>     try :
>>     mpirun -n 1 ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>>     In this scenario, the sequential to sequential vecscatters should
>>     be called.
>>     Then,
>>     mpirun -n 2 ../ex7 -mat_type mpiaijcusparse -vec_type cusp
>>     In this scenario, MPI_General vecscatters should be called ...
>>     and work correctly if you have a system with multiple GPUs.
>>     I
>>     -Paul
>>
>>
>>     On Wed, Jan 22, 2014 at 10:32 AM, Dominic Meiser
>>     <dmeiser at txcorp.com <mailto:dmeiser at txcorp.com>> wrote:
>>
>>         Hey Paul,
>>
>>         Thanks for providing background on this.
>>
>>
>>         On Wed 22 Jan 2014 10:05:13 AM MST, Paul Mullowney wrote:
>>
>>
>>             Dominic,
>>             A few years ago, I was trying to minimize the amount of
>>             data transfer
>>             to and from the GPU (for multi-GPU MatMult) by inspecting
>>             the indices
>>             of the data that needed to be message to and from the
>>             device. Then, I
>>             would call gather kernels on the GPU which pulled the
>>             scattered data
>>             into contiguous buffers and then be transferred to the host
>>             asynchronously (while the MatMult was occurring). The
>>             existence of
>>             VecScatterInitializeForGPU was added in order to build
>>             the necessary
>>             buffers as needed. This was the motivation behind the
>>             existence of
>>             VecScatterInitializeForGPU.
>>             An alternative approach is to message the smallest
>>             contiguous buffer
>>             containing all the data with a single cudaMemcpyAsync.
>>             This is the
>>             method currently implemented.
>>             I never found a case where the former implementation
>>             (with a GPU
>>             gather-kernel) performed better than the alternative
>>             approach which
>>             messaged the smallest contiguous buffer. I looked at
>>             many, many matrices.
>>             Now, as far as I understand the VecScatter kernels, this
>>             method should
>>             only get called if the transfer is MPI_General (i.e. PtoP
>>             parallel to
>>             parallel). Other VecScatter methods are called in other
>>             circumstances
>>             where the the scatter is not MPI_General. That assumption
>>             could be
>>             wrong though.
>>
>>
>>
>>         I see. I figured there was some logic in place to make sure
>>         that this function only gets called in cases where the
>>         transfer type is MPI_General. I'm getting segfaults in this
>>         function where the todata and fromdata are of a different
>>         type. This could easily be user error but I'm not sure. Here
>>         is an example valgrind error:
>>
>>         ==27781== Invalid read of size 8
>>         ==27781== at 0x1188080: VecScatterInitializeForGPU
>>         (vscatcusp.c:46)
>>         ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*,
>>         _p_Vec*, _p_Vec*) (mpiaijcusparse.cu:108
>>         <http://mpiaijcusparse.cu:108>)
>>         ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
>>         ==27781== by 0x4645E4: main (ex7.c:93)
>>         ==27781== Address 0x286305e0 is 1,616 bytes inside a block of
>>         size 1,620 alloc'd
>>         ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
>>         ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int,
>>         char const*, char const*, void**) (mal.c:27)
>>         ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long,
>>         int, char const*, char const*, void**) (mtr.c:186)
>>         ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
>>         ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
>>         ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*,
>>         MatAssemblyType) (mpiaij.c:706)
>>         ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
>>         ==27781== by 0x464301: main (ex7.c:78)
>>
>>         This was produced by src/ksp/ksp/tutorials/ex7.c. The command
>>         line options are
>>
>>         ./ex7 -mat_type mpiaijcusparse -vec_type cusp
>>
>>         In this particular case the todata is of type
>>         VecScatter_Seq_Stride and fromdata is of type
>>         VecScatter_Seq_General. The complete valgrind log (including
>>         configure options for petsc) is attached.
>>
>>         Any comments or suggestions are appreciated.
>>         Cheers,
>>         Dominic
>>
>>
>>             -Paul
>>
>>
>>             On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser
>>             <dmeiser at txcorp.com <mailto:dmeiser at txcorp.com>
>>             <mailto:dmeiser at txcorp.com <mailto:dmeiser at txcorp.com>>>
>>             wrote:
>>
>>             Hi,
>>
>>             I'm trying to understand VecScatterInitializeForGPU in
>>             src/vec/vec/utils/veccusp/__vscatcusp.c. I don't
>>             understand why
>>
>>             this function can get away with casting the fromdata and
>>             todata in
>>             the inctx to VecScatter_MPI_General. Don't we need to
>>             inspect the
>>             VecScatterType fields of the todata and fromdata?
>>
>>             Cheers,
>>             Dominic
>>
>>             --
>>             Dominic Meiser
>>             Tech-X Corporation
>>             5621 Arapahoe Avenue
>>             Boulder, CO 80303
>>             USA
>>             Telephone: 303-996-2036 <tel:303-996-2036>
>>             <tel:303-996-2036 <tel:303-996-2036>>
>>             Fax: 303-448-7756 <tel:303-448-7756> <tel:303-448-7756
>>             <tel:303-448-7756>>
>>             www.txcorp.com <http://www.txcorp.com>
>>             <http://www.txcorp.com>
>>
>>
>>
>>
>>
>>         --
>>         Dominic Meiser
>>         Tech-X Corporation
>>         5621 Arapahoe Avenue
>>         Boulder, CO 80303
>>         USA
>>         Telephone: 303-996-2036 <tel:303-996-2036>
>>         Fax: 303-448-7756 <tel:303-448-7756>
>>         www.txcorp.com <http://www.txcorp.com>
>>
>>
>
>
>     --
>     Dominic Meiser
>     Tech-X Corporation
>     5621 Arapahoe Avenue
>     Boulder, CO 80303
>     USA
>     Telephone:303-996-2036  <tel:303-996-2036>
>     Fax:303-448-7756  <tel:303-448-7756>
>     www.txcorp.com  <http://www.txcorp.com>
>
>



--
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com


From tautges at mcs.anl.gov  Wed Jan 22 12:59:52 2014
From: tautges at mcs.anl.gov (Tim Tautges (ANL))
Date: Wed, 22 Jan 2014 12:59:52 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <CAMYG4GmVRKuy_XryVLcquOAU+0A10pugKWQS2J13rwScb+Mq7w@mail.gmail.com>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>	<52DEE19A.30306@mcs.anl.gov>	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>	<52E00B8E.2020605@mcs.anl.gov>
	<CAMYG4GmVRKuy_XryVLcquOAU+0A10pugKWQS2J13rwScb+Mq7w@mail.gmail.com>
Message-ID: <52E01528.204@mcs.anl.gov>

That's funny, I was thinking the same about DMPlex. :)

- tim

On 01/22/2014 12:45 PM, Matthew Knepley wrote:
> Tim,
>
>    I do not consider MOAB a real alternative here.
>
>       Matt
>
> On Wed, Jan 22, 2014 at 12:18 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>> wrote:
>
>
>
>     On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>
>
>
>                 I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently
>                 sequential, even
>                 when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus
>                 files using
>                 HDF5 as their storage format. Exodus has an parallel extension called nemesis, but I can?t figure out
>                 how how
>                 their concept of ghost point and cells works. The documentation on this point is really unclear.
>
>
>
>         I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel IO front (I?m
>         assuming those guys also run large simulations?). That said, I see this as a two stage process: first integrate with
>         DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of
>         readers/writers. At least this way there will be some agility and interoperability between different formats, and it
>         will not be too disruptive to the application codes when a different formats adopted.
>
>
>     My impression is that the ExodusII people are working within the context of code frameworks more than disk file
>     formats to do this, e.g. in Trilinos and Sierra.  I don't think the ExoII file format by itself is very conducive to
>     representing parallel, which is why Nemesis writes an annotation (though, I haven't followed ExoII developments
>     closely since they went open source several years back).
>
>
>
>
>                 b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching
>                 files ? la
>                 nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an
>                 un-partitionner as a post-processor.
>
>
>         I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry too much
>         about the parallel layout although some clear optimisations are possible. In the  worst case it is a three stage
>         process of where we perform a parallel read of the connectivity, scatter/gather for continuous numbering, parallel
>         repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.
>
>
>     We've seen fragmentation with unstructured meshes being a problem too, and you won't escape that even with
>     renumbering (though reading then migrating would address that, at the cost of some additional communication and
>     possibly reading to figure out where things need to go).
>
>
>
>
>             Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there
>             are other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial)
>             ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness
>             has been
>             iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the
>             process
>             of developing direct support for going between a mesh defined with fields (called tags in MOAB) and petsc
>             vectors.
>             MOAB has pretty solid support for things like computing sharing and ghosting between procs and
>             exchanging/reducing
>             field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the
>             mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a
>             plugin you
>             can enable.  See http://trac.mcs.anl.gov/__projects/ITAPS/wiki/MOAB
>             <http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB> for details of MOAB; the petsc integration stuff
>             is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>
>
>         Another reason this could be of great interest is that MOAB supports (according to the docs) geometric topology
>         which
>         could be used when adapting the mesh on a curved surface for example - another item on my which list.
>
>
>     Yes, MOAB's file format can handle definitions of groupings (and relations between the groups) necessary to
>     represent geometric topology.  What you use for the shape evaluation of those geometric surfaces is another question
>     though.  If you're wanting to do that using a reconstructed continuous patch, MOAB has some code to do that too,
>     though it's not as good as the latest stuff in that area (from Jiao at Stony Brook).
>
>
>
>     Is it
>
>         integrated to PETSc via the plex or does this essentially replace the functionality of the plex?
>
>
>     It's not via plex, but I'm pretty sure all the mesh-related functionality available through plex is available
>     through different API functions in MOAB.
>
>
>     Why does it break
>
>         down for more than 16k procs?
>
>
>     It's a combination of things:
>     - maximizing generality means we're using more than just 2 or 3 tables, because in addition to nodes and elements,
>     we need groupings, whose membership determines whether a group is resident on a given processor, etc, and that
>     strongly affects scaling
>     - that same generality causes us to hit an MPI/IO bug on IBM (though we haven't checked on Q yet to see if that's
>     been addressed, it might have been); we've worked with ANL I/O guys off and on on this, and hope to get back to that
>     on Q soon
>     - we do single file parallel I/O, without any 2-phase (communicate down to I/O nodes then do I/O), and that hits
>     HDF5 pretty hard; we're working with hdfgroup to explore that
>
>     We haven't done any benchmarking on a Lustre system yet, but I expect that to do worse than IBM, because of the many
>     tables thing (my impression is that Lustre doesn't handle frequent metadata reads well)
>
>
>     is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM
>
>         codes such as nek500 that can run on ~1m cores?
>
>
>     Those codes go for a much more restricted I/O data case, which allows them to specialize and do their own
>     implementation of parallel I/O.  So, Nek5000 has its own implementation of poor man's parallel, they repeat vertices
>     in the file that are logically the same (shared between hexes), and they don't really do subsets.  I think that's
>     great to do if you have to, but I'm still hoping for more support in that direction from general libraries.
>
>
>
>             libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel
>             I/O is
>             (but they've been working on it recently I know).
>
>
>
>         Are there any other formats that we should be considering? It?s a few years since I tried playing about with CGNS -
>         at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks interesting as it
>         is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?
>
>
>     Don't know about XDMF.  I know there's been a bunch of work on SILO and its parallel performance fairly recently (3
>     or 4 yrs ago) and it's used heavily inside LLNL.
>
>     - tim
>
>         Cheers Gerard
>
>
>     --
>     ==============================__==============================__====
>     "You will keep in perfect peace him whose mind is
>        steadfast, because he trusts in you."               Isaiah 26:3
>
>                   Tim Tautges            Argonne National Laboratory
>               (tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>)      (telecommuting from UW-Madison)
>       phone (gvoice): (608) 354-1459 <tel:%28608%29%20354-1459>      1500 Engineering Dr.
>                  fax: (608) 263-4499 <tel:%28608%29%20263-4499>      Madison, WI 53706
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any
> results to which their experiments lead.
> -- Norbert Wiener

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706



From knepley at gmail.com  Wed Jan 22 13:00:37 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 22 Jan 2014 13:00:37 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52E01528.204@mcs.anl.gov>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>
	<52DEE19A.30306@mcs.anl.gov>
	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>
	<52E00B8E.2020605@mcs.anl.gov>
	<CAMYG4GmVRKuy_XryVLcquOAU+0A10pugKWQS2J13rwScb+Mq7w@mail.gmail.com>
	<52E01528.204@mcs.anl.gov>
Message-ID: <CAMYG4GmU+7q44jKwr7KWP-sRzRu_924vmz+xq+KNAEuzhDpgwQ@mail.gmail.com>

On Wed, Jan 22, 2014 at 12:59 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov>wrote:

> That's funny, I was thinking the same about DMPlex. :)
>

Maybe you can think that on the moab-dev list.

   Matt


> - tim
>
>
> On 01/22/2014 12:45 PM, Matthew Knepley wrote:
>
>> Tim,
>>
>>    I do not consider MOAB a real alternative here.
>>
>>       Matt
>>
>> On Wed, Jan 22, 2014 at 12:18 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov<mailto:
>> tautges at mcs.anl.gov>> wrote:
>>
>>
>>
>>     On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>>
>>
>>
>>                 I am not sure if Exodus has a good solution here. As far
>> as I understand, exodus is inherently
>>                 sequential, even
>>                 when implemented with HDF5 instead of netcdf. I would
>> also worry about third party support for exodus
>>                 files using
>>                 HDF5 as their storage format. Exodus has an parallel
>> extension called nemesis, but I can?t figure out
>>                 how how
>>                 their concept of ghost point and cells works. The
>> documentation on this point is really unclear.
>>
>>
>>
>>         I have to admit I was kind of hoping that ExodusII folks would
>> have come on a bit more on the parallel IO front (I?m
>>         assuming those guys also run large simulations?). That said, I
>> see this as a two stage process: first integrate with
>>         DMPlex as that should give the high level abstraction for
>> read/write to file; secondly extend the family of
>>         readers/writers. At least this way there will be some agility and
>> interoperability between different formats, and it
>>         will not be too disruptive to the application codes when a
>> different formats adopted.
>>
>>
>>     My impression is that the ExodusII people are working within the
>> context of code frameworks more than disk file
>>     formats to do this, e.g. in Trilinos and Sierra.  I don't think the
>> ExoII file format by itself is very conducive to
>>     representing parallel, which is why Nemesis writes an annotation
>> (though, I haven't followed ExoII developments
>>     closely since they went open source several years back).
>>
>>
>>
>>
>>                 b. Do ?poor man? parallel I/O where each CPU does its own
>> I/O, and possibly create interface matching
>>                 files ? la
>>                 nemesis or SILO. Maybe, we can save enough information on
>> the parallel layout in order to easily write an
>>                 un-partitionner as a post-processor.
>>
>>
>>         I am pretty sure that if we are writing everything in slabs to a
>> HDF5 container we do not have to worry too much
>>         about the parallel layout although some clear optimisations are
>> possible. In the  worst case it is a three stage
>>         process of where we perform a parallel read of the connectivity,
>> scatter/gather for continuous numbering, parallel
>>         repartitioning and subsequent parallel read of remaining data.
>> Importantly, it is at least scalable.
>>
>>
>>     We've seen fragmentation with unstructured meshes being a problem
>> too, and you won't escape that even with
>>     renumbering (though reading then migrating would address that, at the
>> cost of some additional communication and
>>     possibly reading to figure out where things need to go).
>>
>>
>>
>>
>>             Depending on the degree of direct interaction/automation in
>> those interactions between the mesh and Petsc, there
>>             are other options as well.  One that we're developing, based
>> on the MOAB library, can read/write (in serial)
>>             ExodusII, and also supports parallel read/write using its own
>> HDF5-based format.  Parallel I/O robustness
>>             has been
>>             iffy above ~16k procs and 32M-64M hex/tet elements, but for
>> smaller problems it should work.  We're in the
>>             process
>>             of developing direct support for going between a mesh defined
>> with fields (called tags in MOAB) and petsc
>>             vectors.
>>             MOAB has pretty solid support for things like computing
>> sharing and ghosting between procs and
>>             exchanging/reducing
>>             field values on those entities.  Viz is supported either by
>> compiling a VTK/Paraview plugin that pulls the
>>             mesh/fields through MOAB or by translating to VTK (also
>> supported directly from MOAB); Visit also has a
>>             plugin you
>>             can enable.  See http://trac.mcs.anl.gov/__
>> projects/ITAPS/wiki/MOAB
>>
>>             <http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB> for
>> details of MOAB; the petsc integration stuff
>>             is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>>
>>
>>         Another reason this could be of great interest is that MOAB
>> supports (according to the docs) geometric topology
>>         which
>>         could be used when adapting the mesh on a curved surface for
>> example - another item on my which list.
>>
>>
>>     Yes, MOAB's file format can handle definitions of groupings (and
>> relations between the groups) necessary to
>>     represent geometric topology.  What you use for the shape evaluation
>> of those geometric surfaces is another question
>>     though.  If you're wanting to do that using a reconstructed
>> continuous patch, MOAB has some code to do that too,
>>     though it's not as good as the latest stuff in that area (from Jiao
>> at Stony Brook).
>>
>>
>>
>>     Is it
>>
>>         integrated to PETSc via the plex or does this essentially replace
>> the functionality of the plex?
>>
>>
>>     It's not via plex, but I'm pretty sure all the mesh-related
>> functionality available through plex is available
>>     through different API functions in MOAB.
>>
>>
>>     Why does it break
>>
>>         down for more than 16k procs?
>>
>>
>>     It's a combination of things:
>>     - maximizing generality means we're using more than just 2 or 3
>> tables, because in addition to nodes and elements,
>>     we need groupings, whose membership determines whether a group is
>> resident on a given processor, etc, and that
>>     strongly affects scaling
>>     - that same generality causes us to hit an MPI/IO bug on IBM (though
>> we haven't checked on Q yet to see if that's
>>     been addressed, it might have been); we've worked with ANL I/O guys
>> off and on on this, and hope to get back to that
>>     on Q soon
>>     - we do single file parallel I/O, without any 2-phase (communicate
>> down to I/O nodes then do I/O), and that hits
>>     HDF5 pretty hard; we're working with hdfgroup to explore that
>>
>>     We haven't done any benchmarking on a Lustre system yet, but I expect
>> that to do worse than IBM, because of the many
>>     tables thing (my impression is that Lustre doesn't handle frequent
>> metadata reads well)
>>
>>
>>     is it just a case that Lustre gets hammered? What magic sauce is used
>> by high order FEM
>>
>>         codes such as nek500 that can run on ~1m cores?
>>
>>
>>     Those codes go for a much more restricted I/O data case, which allows
>> them to specialize and do their own
>>     implementation of parallel I/O.  So, Nek5000 has its own
>> implementation of poor man's parallel, they repeat vertices
>>     in the file that are logically the same (shared between hexes), and
>> they don't really do subsets.  I think that's
>>     great to do if you have to, but I'm still hoping for more support in
>> that direction from general libraries.
>>
>>
>>
>>             libmesh also maintains its own DMlibmesh, but I'm not sure
>> how solid their support for large mesh / parallel
>>             I/O is
>>             (but they've been working on it recently I know).
>>
>>
>>
>>         Are there any other formats that we should be considering? It?s a
>> few years since I tried playing about with CGNS -
>>         at the time its parallel IO was non-existent and I have not seen
>> it being pushed since. XDMF looks interesting as it
>>         is essentially some xml metadata and a HDF5 bucket. Is anyone
>> championing this?
>>
>>
>>     Don't know about XDMF.  I know there's been a bunch of work on SILO
>> and its parallel performance fairly recently (3
>>     or 4 yrs ago) and it's used heavily inside LLNL.
>>
>>     - tim
>>
>>         Cheers Gerard
>>
>>
>>     --
>>     ==============================__==============================__====
>>
>>     "You will keep in perfect peace him whose mind is
>>        steadfast, because he trusts in you."               Isaiah 26:3
>>
>>                   Tim Tautges            Argonne National Laboratory
>>               (tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>)
>>  (telecommuting from UW-Madison)
>>       phone (gvoice): (608) 354-1459 <tel:%28608%29%20354-1459>
>>  1500 Engineering Dr.
>>                  fax: (608) 263-4499 <tel:%28608%29%20263-4499>
>>  Madison, WI 53706
>>
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any
>> results to which their experiments lead.
>> -- Norbert Wiener
>>
>
> --
> ================================================================
> "You will keep in perfect peace him whose mind is
>   steadfast, because he trusts in you."               Isaiah 26:3
>
>              Tim Tautges            Argonne National Laboratory
>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
>             fax: (608) 263-4499      Madison, WI 53706
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/53f2314e/attachment.html>

From tautges at mcs.anl.gov  Wed Jan 22 13:06:00 2014
From: tautges at mcs.anl.gov (Tim Tautges (ANL))
Date: Wed, 22 Jan 2014 13:06:00 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <CAMYG4GmU+7q44jKwr7KWP-sRzRu_924vmz+xq+KNAEuzhDpgwQ@mail.gmail.com>
References: <mailman.29.1390327205.15233.petsc-dev@mcs.anl.gov>	<52DEE19A.30306@mcs.anl.gov>	<2A0CC016-8D90-4BE4-84CC-5804DA045291@imperial.ac.uk>	<52E00B8E.2020605@mcs.anl.gov>	<CAMYG4GmVRKuy_XryVLcquOAU+0A10pugKWQS2J13rwScb+Mq7w@mail.gmail.com>	<52E01528.204@mcs.anl.gov>
	<CAMYG4GmU+7q44jKwr7KWP-sRzRu_924vmz+xq+KNAEuzhDpgwQ@mail.gmail.com>
Message-ID: <52E01698.5090702@mcs.anl.gov>

Ok, message received.  Later.

- tim

On 01/22/2014 01:00 PM, Matthew Knepley wrote:
> On Wed, Jan 22, 2014 at 12:59 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>> wrote:
>
>     That's funny, I was thinking the same about DMPlex. :)
>
>
> Maybe you can think that on the moab-dev list.
>
>     Matt
>
>     - tim
>
>
>     On 01/22/2014 12:45 PM, Matthew Knepley wrote:
>
>         Tim,
>
>             I do not consider MOAB a real alternative here.
>
>                Matt
>
>         On Wed, Jan 22, 2014 at 12:18 PM, Tim Tautges (ANL) <tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>
>         <mailto:tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>>> wrote:
>
>
>
>              On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>
>
>
>                          I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently
>                          sequential, even
>                          when implemented with HDF5 instead of netcdf. I would also worry about third party support for
>         exodus
>                          files using
>                          HDF5 as their storage format. Exodus has an parallel extension called nemesis, but I can?t
>         figure out
>                          how how
>                          their concept of ghost point and cells works. The documentation on this point is really unclear.
>
>
>
>                  I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel
>         IO front (I?m
>                  assuming those guys also run large simulations?). That said, I see this as a two stage process: first
>         integrate with
>                  DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of
>                  readers/writers. At least this way there will be some agility and interoperability between different
>         formats, and it
>                  will not be too disruptive to the application codes when a different formats adopted.
>
>
>              My impression is that the ExodusII people are working within the context of code frameworks more than disk file
>              formats to do this, e.g. in Trilinos and Sierra.  I don't think the ExoII file format by itself is very
>         conducive to
>              representing parallel, which is why Nemesis writes an annotation (though, I haven't followed ExoII developments
>              closely since they went open source several years back).
>
>
>
>
>                          b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface
>         matching
>                          files ? la
>                          nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to
>         easily write an
>                          un-partitionner as a post-processor.
>
>
>                  I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry
>         too much
>                  about the parallel layout although some clear optimisations are possible. In the  worst case it is a
>         three stage
>                  process of where we perform a parallel read of the connectivity, scatter/gather for continuous
>         numbering, parallel
>                  repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.
>
>
>              We've seen fragmentation with unstructured meshes being a problem too, and you won't escape that even with
>              renumbering (though reading then migrating would address that, at the cost of some additional communication and
>              possibly reading to figure out where things need to go).
>
>
>
>
>                      Depending on the degree of direct interaction/automation in those interactions between the mesh and
>         Petsc, there
>                      are other options as well.  One that we're developing, based on the MOAB library, can read/write
>         (in serial)
>                      ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O
>         robustness
>                      has been
>                      iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're
>         in the
>                      process
>                      of developing direct support for going between a mesh defined with fields (called tags in MOAB) and
>         petsc
>                      vectors.
>                      MOAB has pretty solid support for things like computing sharing and ghosting between procs and
>                      exchanging/reducing
>                      field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that
>         pulls the
>                      mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a
>                      plugin you
>                      can enable.  See http://trac.mcs.anl.gov/____projects/ITAPS/wiki/MOAB
>         <http://trac.mcs.anl.gov/__projects/ITAPS/wiki/MOAB>
>
>                      <http://trac.mcs.anl.gov/__projects/ITAPS/wiki/MOAB
>         <http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB>> for details of MOAB; the petsc integration stuff
>                      is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>
>
>                  Another reason this could be of great interest is that MOAB supports (according to the docs) geometric
>         topology
>                  which
>                  could be used when adapting the mesh on a curved surface for example - another item on my which list.
>
>
>              Yes, MOAB's file format can handle definitions of groupings (and relations between the groups) necessary to
>              represent geometric topology.  What you use for the shape evaluation of those geometric surfaces is another
>         question
>              though.  If you're wanting to do that using a reconstructed continuous patch, MOAB has some code to do that
>         too,
>              though it's not as good as the latest stuff in that area (from Jiao at Stony Brook).
>
>
>
>              Is it
>
>                  integrated to PETSc via the plex or does this essentially replace the functionality of the plex?
>
>
>              It's not via plex, but I'm pretty sure all the mesh-related functionality available through plex is available
>              through different API functions in MOAB.
>
>
>              Why does it break
>
>                  down for more than 16k procs?
>
>
>              It's a combination of things:
>              - maximizing generality means we're using more than just 2 or 3 tables, because in addition to nodes and
>         elements,
>              we need groupings, whose membership determines whether a group is resident on a given processor, etc, and that
>              strongly affects scaling
>              - that same generality causes us to hit an MPI/IO bug on IBM (though we haven't checked on Q yet to see if
>         that's
>              been addressed, it might have been); we've worked with ANL I/O guys off and on on this, and hope to get
>         back to that
>              on Q soon
>              - we do single file parallel I/O, without any 2-phase (communicate down to I/O nodes then do I/O), and that
>         hits
>              HDF5 pretty hard; we're working with hdfgroup to explore that
>
>              We haven't done any benchmarking on a Lustre system yet, but I expect that to do worse than IBM, because of
>         the many
>              tables thing (my impression is that Lustre doesn't handle frequent metadata reads well)
>
>
>              is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM
>
>                  codes such as nek500 that can run on ~1m cores?
>
>
>              Those codes go for a much more restricted I/O data case, which allows them to specialize and do their own
>              implementation of parallel I/O.  So, Nek5000 has its own implementation of poor man's parallel, they repeat
>         vertices
>              in the file that are logically the same (shared between hexes), and they don't really do subsets.  I think
>         that's
>              great to do if you have to, but I'm still hoping for more support in that direction from general libraries.
>
>
>
>                      libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh /
>         parallel
>                      I/O is
>                      (but they've been working on it recently I know).
>
>
>
>                  Are there any other formats that we should be considering? It?s a few years since I tried playing about
>         with CGNS -
>                  at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks
>         interesting as it
>                  is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?
>
>
>              Don't know about XDMF.  I know there's been a bunch of work on SILO and its parallel performance fairly
>         recently (3
>              or 4 yrs ago) and it's used heavily inside LLNL.
>
>              - tim
>
>                  Cheers Gerard
>
>
>              --
>              ==============================____============================__==__====
>
>              "You will keep in perfect peace him whose mind is
>                 steadfast, because he trusts in you."               Isaiah 26:3
>
>                            Tim Tautges            Argonne National Laboratory
>                        (tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov> <mailto:tautges at mcs.anl.gov
>         <mailto:tautges at mcs.anl.gov>>)      (telecommuting from UW-Madison)
>                phone (gvoice): (608) 354-1459 <tel:%28608%29%20354-1459> <tel:%28608%29%20354-1459>      1500
>         Engineering Dr.
>                           fax: (608) 263-4499 <tel:%28608%29%20263-4499> <tel:%28608%29%20263-4499>      Madison, WI 53706
>
>
>
>
>
>         --
>         What most experimenters take for granted before they begin their experiments is infinitely more interesting than any
>         results to which their experiments lead.
>         -- Norbert Wiener
>
>
>     --
>     ==============================__==============================__====
>     "You will keep in perfect peace him whose mind is
>        steadfast, because he trusts in you."               Isaiah 26:3
>
>                   Tim Tautges            Argonne National Laboratory
>               (tautges at mcs.anl.gov <mailto:tautges at mcs.anl.gov>)      (telecommuting from UW-Madison)
>       phone (gvoice): (608) 354-1459 <tel:%28608%29%20354-1459>      1500 Engineering Dr.
>                  fax: (608) 263-4499 <tel:%28608%29%20263-4499>      Madison, WI 53706
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any
> results to which their experiments lead.
> -- Norbert Wiener

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706



From knepley at gmail.com  Wed Jan 22 13:09:23 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 22 Jan 2014 13:09:23 -0600
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
Message-ID: <CAMYG4GkDBxWW4=Kt-K5xP285gX8es2N7rax-vE-5ViS6W21_6Q@mail.gmail.com>

On Tue, Jan 21, 2014 at 11:01 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Jan 21, 2014 at 9:30 AM, Blaise A Bourdin <bourdin at lsu.edu> wrote:
>
>> Hi,
>>
>> It looks like DMplex is steadily gaining maturity but I/O is lagging
>> behind. As far as I understand, right now, PETSc can _read_ a mesh in
>> exodus format, and  write binary VTS format, but many issues remain, IMHO:
>>    - The exodus reader relies on a hard-coded nodeset named ?marker?.
>> Generating such a nodeset is not trivial
>>      (at least not for complex meshes generated with Cubit / Trelis).
>>
>
> I will fix this right away. I will put in some registration mechanism for
> labels, and we can iterate.
>

I just looked at the code again, and this is not what happens. The exodus
reader reads all cell, vertex, and side
sets. What you are remembering is that when I use mesh generators
(Triangle, TetGen) I call the boundary
markers from their output "marker". Thus there should be no problem. I will
start working on an exodus writer.

I think we should have an exodus writer, since the format handles the
metadata well, and then have our own
HDF5 format since we can write in parallel, use Xdmf, and it will be
extensible by us to handle different discretizations.

   Matt



>    - Reading from or writing to exodus files is not supported.
>>
>
> Yes, I think this is the best target. It should be similar to writing HDF5
> that we do for PyLith.
>
>    Matt
>
>
>>    - The VTS viewer only allows to read and write _all_ fields in a DM.
>> This may be overkill if one only
>>      wants to read boundary values, for instance.
>>    - The VTS viewer loses all informations on exodus nodesets and cell
>> sets. These may have some significance
>>      and may be required to exploit the output of a computations.
>>    - VTS seems to have a concept of ?blocks?. My understanding is that
>> the parallel VTS viewer uses blocks to
>>      save subdomains, and that continuity of piecewise linear fields
>> across subdomain boundaries is lost.
>>      It is not entirely clear to me if with this layout, it would be
>> possible to reopen a file with a
>>      different processor count.
>>
>> I can dedicate some resources to improving DMplex I/O. Perhaps we can
>> start a discussion by listing the desired features such readers / writers
>> should have. I will pitch in by listing what matters to me:
>>    - A well documented and adopted file format that most post-processors
>> / visualization tools can use
>>    - Ability to read / write individual fields
>>    - Preserve _all_ information from the exodus file (node / side / cell
>> sets), do not lose continuity of fields
>>      across subdomain boundaries.
>>    - Ability to reopen file on a different cpu count
>>    - Support for higher order elements
>>
>> Am I missing something? If not, we can follow up with discussion on
>> formats and implementation.
>>
>> Blaise
>>
>> --
>> Department of Mathematics and Center for Computation & Technology
>> Louisiana State University, Baton Rouge, LA 70803, USA
>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>> http://www.math.lsu.edu/~bourdin
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/995ac3e4/attachment.html>

From bsmith at mcs.anl.gov  Wed Jan 22 13:16:27 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 22 Jan 2014 13:16:27 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <8738kfexe1.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org> <871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
Message-ID: <7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>


  When we first tried to follow ?community standards? with ?prefix I was told that ?community standards? did not recognize the need for multiple installs and that community standards did not allow for putting archs onto names to allow multiple installs.  We specifically removed the concept of PETSC_ARCH for installs for this reason. Have community standards changed?

  Barry


    


On Jan 22, 2014, at 11:31 AM, Jed Brown <jed at jedbrown.org> wrote:

> Sean Farley <sean.michael.farley at gmail.com> writes:
> 
>> jed at jedbrown.org writes:
>> 
>>> Sean Farley <sean.michael.farley at gmail.com> writes:
>>>>>  $prefix/bin/petsc-${version}-${arch}-config
>>>>> 
>>>>> and symlink $prefix/bin/petsc-config to it by default?
>>>>> 
>>>>> Then always namespace libraries and includes?
>>>> 
>>>> This is what I had in mind as well. Though, distributions would remove
>>>> the symlink since that would conflict.
>>> 
>>> Debian would probably manage it through /etc/alternatives.
>> 
>> Just a note that MacPorts would manage it through etc/select :-)
> 
> Thanks, Sean.  (I really was asking you as much as Barry/Matt/Satish/etc.
> Packaging is a different environment.)
> 
> What do the rest of you think?



From jed at jedbrown.org  Wed Jan 22 13:24:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 12:24:58 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org> <87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
Message-ID: <87d2jjbz05.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>   When we first tried to follow ?community standards? with ?prefix I
>   was told that ?community standards? did not recognize the need for
>   multiple installs and that community standards did not allow for
>   putting archs onto names to allow multiple installs.  We
>   specifically removed the concept of PETSC_ARCH for installs for this
>   reason. Have community standards changed?

There is a big difference between having the install variant in the path
(user has to manage paths themselves) and versioning libraries installed
to the same path.  We can continue with using the prefix for everything
(which some distros will work around to avoid offloading path management
to the user) or we can allow multiple installs to a standard path (users
don't have to deal with paths, instead run petsc-3.5.0-dbg-config).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/e89a6127/attachment.sig>

From sean.michael.farley at gmail.com  Wed Jan 22 13:27:38 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Wed, 22 Jan 2014 13:27:38 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org> <871u0617dt.fsf@jedbrown.org>
	<0E818089-5352-4850-9AC9-A0E05921D346@mcs.anl.gov>
	<87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
Message-ID: <m2a9enbyvp.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

>   When we first tried to follow ?community standards? with ?prefix I was told that ?community standards? did not recognize the need for multiple installs and that community standards did not allow for putting archs onto names to allow multiple installs.  We specifically removed the concept of PETSC_ARCH for installs for this reason. Have community standards changed?

There are a lot more (convenient?) flags that allow subprefix changing:
--bindir, --libdir, --includedir, --docdir, --mandir, etc. that allow a
similar concept to PETSC_ARCH.


From bsmith at mcs.anl.gov  Wed Jan 22 13:38:09 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 22 Jan 2014 13:38:09 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <87d2jjbz05.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org> <87d2jqymrw.fsf@jedbrown.org>
	<0C0B29BE-CC58-4A70-8030-52FF72BF260E@mcs.anl.gov>
	<874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
	<87d2jjbz05.fsf@jedbrown.org>
Message-ID: <9E9B8294-12B1-470C-8F32-2587A63429ED@mcs.anl.gov>


On Jan 22, 2014, at 1:24 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>  When we first tried to follow ?community standards? with ?prefix I
>>  was told that ?community standards? did not recognize the need for
>>  multiple installs and that community standards did not allow for
>>  putting archs onto names to allow multiple installs.  We
>>  specifically removed the concept of PETSC_ARCH for installs for this
>>  reason. Have community standards changed?
> 
> There is a big difference between having the install variant in the path

   I have no idea what the sentence fragment above is suppose to mean. 

> (user has to manage paths themselves) and versioning libraries installed
> to the same path.  We can continue with using the prefix for everything
> (which some distros will work around to avoid offloading path management
> to the user) or we can allow multiple installs to a standard path (users
> don't have to deal with paths, instead run petsc-3.5.0-dbg-config).

   I am not fundamentally objecting to name spacing the things installed but I previously understood that this was considered a big no-no.

   Barry




From jed at jedbrown.org  Wed Jan 22 13:43:46 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 12:43:46 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <9E9B8294-12B1-470C-8F32-2587A63429ED@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org> <874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
	<87d2jjbz05.fsf@jedbrown.org>
	<9E9B8294-12B1-470C-8F32-2587A63429ED@mcs.anl.gov>
Message-ID: <877g9rby4t.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

> On Jan 22, 2014, at 1:24 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>> 
>>>  When we first tried to follow ?community standards? with ?prefix I
>>>  was told that ?community standards? did not recognize the need for
>>>  multiple installs and that community standards did not allow for
>>>  putting archs onto names to allow multiple installs.  We
>>>  specifically removed the concept of PETSC_ARCH for installs for this
>>>  reason. Have community standards changed?
>> 
>> There is a big difference between having the install variant in the path
>
>    I have no idea what the sentence fragment above is suppose to mean. 

/opt/petsc/arch-dbg/

User has to manually set flags to find that path.  If we didn't set
RPATH, they'd also need LD_LIBRARY_PATH.  (Not setting RPATH is nice for
testing new system configurations, among other things.)

>> (user has to manage paths themselves) and versioning libraries installed
>> to the same path.  We can continue with using the prefix for everything
>> (which some distros will work around to avoid offloading path management
>> to the user) or we can allow multiple installs to a standard path (users
>> don't have to deal with paths, instead run petsc-3.5.0-dbg-config).
>
>    I am not fundamentally objecting to name spacing the things
>    installed but I previously understood that this was considered a
>    big no-no.

Installing to funky paths is discouraged, but versioning libraries is
encouraged.  If there is only one variant, that versioning is usually
libfoo.so.1.2.3.  Then you can have several versions installed and
precompiled binaries can keep using the old versions.

When there are multiple variants (e.g, python2 and python3), the
libraries usually absorb the variant name.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/35620f69/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan 22 13:54:36 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 22 Jan 2014 13:54:36 -0600
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <877g9rby4t.fsf@jedbrown.org>
References: <87zjmu1ove.fsf@jedbrown.org> <874n51xmts.fsf@jedbrown.org>
	<DA29A82B-C091-4306-B9D1-42A1770B0720@mcs.anl.gov>
	<87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
	<87d2jjbz05.fsf@jedbrown.org>
	<9E9B8294-12B1-470C-8F32-2587A63429ED@mcs.anl.gov>
	<877g9rby4t.fsf@jedbrown.org>
Message-ID: <4147EE7C-2404-45E0-B1DF-4CD72A4EEF1A@mcs.anl.gov>


  I have no objection to someone setting up a branch to test allowing versioning (and PETSC_ARCH) with the ?prefix option.

   Barry

On Jan 22, 2014, at 1:43 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>> On Jan 22, 2014, at 1:24 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>> 
>>>> When we first tried to follow ?community standards? with ?prefix I
>>>> was told that ?community standards? did not recognize the need for
>>>> multiple installs and that community standards did not allow for
>>>> putting archs onto names to allow multiple installs.  We
>>>> specifically removed the concept of PETSC_ARCH for installs for this
>>>> reason. Have community standards changed?
>>> 
>>> There is a big difference between having the install variant in the path
>> 
>>   I have no idea what the sentence fragment above is suppose to mean. 
> 
> /opt/petsc/arch-dbg/
> 
> User has to manually set flags to find that path.  If we didn't set
> RPATH, they'd also need LD_LIBRARY_PATH.  (Not setting RPATH is nice for
> testing new system configurations, among other things.)
> 
>>> (user has to manage paths themselves) and versioning libraries installed
>>> to the same path.  We can continue with using the prefix for everything
>>> (which some distros will work around to avoid offloading path management
>>> to the user) or we can allow multiple installs to a standard path (users
>>> don't have to deal with paths, instead run petsc-3.5.0-dbg-config).
>> 
>>   I am not fundamentally objecting to name spacing the things
>>   installed but I previously understood that this was considered a
>>   big no-no.
> 
> Installing to funky paths is discouraged, but versioning libraries is
> encouraged.  If there is only one variant, that versioning is usually
> libfoo.so.1.2.3.  Then you can have several versions installed and
> precompiled binaries can keep using the old versions.
> 
> When there are multiple variants (e.g, python2 and python3), the
> libraries usually absorb the variant name.



From irving at naml.us  Wed Jan 22 14:01:31 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 22 Jan 2014 12:01:31 -0800
Subject: [petsc-dev] the non-implemented case of PetscFEIntegrateResidual et
	al.
Message-ID: <CAJ1ofpf5vpQ0wCHr8KvCFa40GE=bj1jvwos4xL9ifgCajpts_w@mail.gmail.com>

If a PetscFE object doesn't define integrateresidual, a call to
PetscFEIntegrateResidual silently does nothing.  Is this intended
behavior, or would it be better to complain similar to what happens
for an undefined matrix operation?  I ask because I'm about to add an
operation (PetscFEIntegrateScalars) which will initially only be
defined in basic mode.

Geoffrey


From knepley at gmail.com  Wed Jan 22 14:06:26 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 22 Jan 2014 14:06:26 -0600
Subject: [petsc-dev] the non-implemented case of
 PetscFEIntegrateResidual et al.
In-Reply-To: <CAJ1ofpf5vpQ0wCHr8KvCFa40GE=bj1jvwos4xL9ifgCajpts_w@mail.gmail.com>
References: <CAJ1ofpf5vpQ0wCHr8KvCFa40GE=bj1jvwos4xL9ifgCajpts_w@mail.gmail.com>
Message-ID: <CAMYG4G=ynBmbtdybX7gpDjZB=a0FR69f1_PQ1w08N357F=OAyA@mail.gmail.com>

On Wed, Jan 22, 2014 at 2:01 PM, Geoffrey Irving <irving at naml.us> wrote:

> If a PetscFE object doesn't define integrateresidual, a call to
> PetscFEIntegrateResidual silently does nothing.  Is this intended
> behavior, or would it be better to complain similar to what happens
> for an undefined matrix operation?  I ask because I'm about to add an
> operation (PetscFEIntegrateScalars) which will initially only be
> defined in basic mode.


You are correct, it should error. THis is left over from me building it.

  Thanks,

     Matt


>
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/7edb0c0b/attachment.html>

From jed at jedbrown.org  Wed Jan 22 14:09:16 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 13:09:16 -0700
Subject: [petsc-dev] CFLAGS and COPTFLAGS
In-Reply-To: <4147EE7C-2404-45E0-B1DF-4CD72A4EEF1A@mcs.anl.gov>
References: <87zjmu1ove.fsf@jedbrown.org> <87eh43u48c.fsf@jedbrown.org>
	<234B1E1E-E31F-4393-8BC5-E10F10C22289@mcs.anl.gov>
	<87vbxfr0ny.fsf@jedbrown.org>
	<5E896FE1-256B-4A86-9EFA-0E8AE0F99BAD@mcs.anl.gov>
	<m2a9ercx4f.fsf@gmail.com>
	<alpine.LFD.2.11.1401191833450.7308@asterix>
	<B6D89C9C-979A-4F76-BB73-5D1CA04E959C@mcs.anl.gov>
	<8738kjpbyy.fsf@jedbrown.org> <m2zjmqbi9a.fsf@gmail.com>
	<25EC669B-4D4D-470A-BE3E-EB1EB9D7F7AE@mcs.anl.gov>
	<87bnz6l9ac.fsf@jedbrown.org> <m2eh40at0d.fsf@gmail.com>
	<87wqhsdktv.fsf@jedbrown.org> <m2d2jkaq3x.fsf@gmail.com>
	<8738kfexe1.fsf@jedbrown.org>
	<7DD261A1-A8BB-4D30-804A-5B23630B81CE@mcs.anl.gov>
	<87d2jjbz05.fsf@jedbrown.org>
	<9E9B8294-12B1-470C-8F32-2587A63429ED@mcs.anl.gov>
	<877g9rby4t.fsf@jedbrown.org>
	<4147EE7C-2404-45E0-B1DF-4CD72A4EEF1A@mcs.anl.gov>
Message-ID: <87y527aidv.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>   I have no objection to someone setting up a branch to test allowing
>   versioning (and PETSC_ARCH) with the ?prefix option.

Okay, adding to my "todo".
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/e37aa258/attachment.sig>

From irving at naml.us  Wed Jan 22 14:29:17 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 22 Jan 2014 12:29:17 -0800
Subject: [petsc-dev] questions about PetscFEIntegrateResidual_Basic
In-Reply-To: <CAJ1ofpeCci_SEB2h_AmwLKiZfgA0j_j=-kZSawtTzf11vC+xcA@mail.gmail.com>
References: <CAJ1ofpeCci_SEB2h_AmwLKiZfgA0j_j=-kZSawtTzf11vC+xcA@mail.gmail.com>
Message-ID: <CAJ1ofpfrvfFsFepkwv1=i5qNozJp2AcKfSaiM13tn-53JH4vTg@mail.gmail.com>

On Tue, Jan 21, 2014 at 10:12 AM, Geoffrey Irving <irving at naml.us> wrote:
> PetscFEIntegrateResidual_Basic seems to have a redundant argument
> list.  It takes a single PetscFE, an array of PetscFE's, and a field
> index into the array.  The single PetscFE argument is ignored.  I
> would imagine that either the field index or the single PetscFE should
> be eliminated.
>
> Also, PetscFEIntegrateResidual_Basic allocates the f0 array with size
>
>     quad.numPoints * (spatial dimension of fe[0])
>
> but accesses it as if it had size
>
>     quad.numPoints * (number of components of fe[field])
>
> Am I reading this right?

Correction: the single PetscFE is currently used to look up
integration routines, so is far from unused.  However, it might be
possible to look up these routines in the other PetscFE objects, and
we might also want to verify that all PetscFE's have the same type to
avoid corrupt behavior (not sure if that would be a problem or not).

I still think we allocate f0 as the wrong size, though.

Geoffrey


From rupp at mcs.anl.gov  Wed Jan 22 15:34:26 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 22:34:26 +0100
Subject: [petsc-dev] VecScatterInitializeForGPU
In-Reply-To: <52E00098.4050601@txcorp.com>
References: <52DFF67C.5000504@txcorp.com>	<CAMJ8fwqR0=v0dodahCEAhcfhgOBWyjSVdOZfSbd+jjmLeadfHA@mail.gmail.com>
	<52E00098.4050601@txcorp.com>
Message-ID: <52E03962.2050503@mcs.anl.gov>

Hi guys,

 > I see. I figured there was some logic in place to make sure that this
> function only gets called in cases where the transfer type is
> MPI_General. I'm getting segfaults in this function where the todata and
> fromdata are of a different type. This could easily be user error but
> I'm not sure. Here is an example valgrind error:
>
> ==27781== Invalid read of size 8
> ==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
> ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*)
> (mpiaijcusparse.cu:108)
> ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
> ==27781== by 0x4645E4: main (ex7.c:93)
> ==27781== Address 0x286305e0 is 1,616 bytes inside a block of size 1,620
> alloc'd
> ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
> ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*,
> char const*, void**) (mal.c:27)
> ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char
> const*, char const*, void**) (mtr.c:186)
> ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
> ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
> ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType)
> (mpiaij.c:706)
> ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
> ==27781== by 0x464301: main (ex7.c:78)

Okay, so this is another problem related to VecScatter* in addition to 
the ones reported for mpiaijcusp matrices these days. It seems like 
something is fundamentally broken with vector scatters for GPU data 
types right now. :-(

Best regards,
Karli



From rupp at mcs.anl.gov  Wed Jan 22 15:43:53 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 22 Jan 2014 22:43:53 +0100
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <CAJzU6sCEDeyHFnTKU4zc+zOhqpbL77GvKzucVCwTOeGAFaHyJg@mail.gmail.com>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>	<52DE3DBC.80000@mcs.anl.gov>	<CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>	<52DEE695.5080200@mcs.anl.gov>
	<CAJzU6sCEDeyHFnTKU4zc+zOhqpbL77GvKzucVCwTOeGAFaHyJg@mail.gmail.com>
Message-ID: <52E03B99.3050604@mcs.anl.gov>

Hi Mani,

 > Thanks Karl. Is there anyway I can get the build info?

For the moment only in case of an error...


> Also, my system is such that I have the GPU on one platform (0) and the
> CPU on another (1). When I try using -viennalcl_device_cpu, it still
> uses the GPU cause I think that it defaults to the first platform it
> finds which only has the GPU. Is there some way to toggle between the
> platforms or do you suggest that I manually pass in the contexts into
> viennacl?

Ah, so you have two OpenCL platforms available on your machine. Call
   viennacl::ocl::set_context_platform_index(0, platform_index_here);
at the beginning of your program in order to specify the default 
platform index. In your case this should be
   viennacl::ocl::set_context_platform_index(0, 1);
(cf. http://viennacl.sourceforge.net/viennacl-manual-current.pdf)
I'll also make this configurable directly through PETSc in the next 
update of the ViennaCL bindings.

Best regards,
Karli



From baagaard at usgs.gov  Wed Jan 22 15:51:22 2014
From: baagaard at usgs.gov (Brad Aagaard)
Date: Wed, 22 Jan 2014 13:51:22 -0800
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <87wqhrdimj.fsf@jedbrown.org>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org> <52DFFCE4.1020503@usgs.gov>
	<87wqhrdimj.fsf@jedbrown.org>
Message-ID: <52E03D5A.1020906@usgs.gov>

On 01/22/2014 09:35 AM, Jed Brown wrote:
> Brad Aagaard <baagaard at usgs.gov> writes:
>> We have plans to add higher order basis functions to PyLith this spring,
>> and recognize the deficiencies in many of the available formats. It
>> would be nice to have a consistent way to deal with DMPlex I/O and these
>> other issues (boundaries, high order, etc) across various codes using
>> DMPlex.
>
> How important is it to be able to visualize "checkpoint" files, or to
> restart from "visualization" files?  (Many applications write totally
> different files because there is no way to include the semantic
> information in vis files.)

I agree that one would probably not visualize "checkpoint" files nor 
restart from "visualization" files. I wasn't able to follow where the 
discussion separated these issues. Is the DMPlex I/O targeting 
checkpointing and ExodusII/HDF5/etc targeting viz?

Brad



From jed at jedbrown.org  Wed Jan 22 16:07:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 22 Jan 2014 15:07:58 -0700
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <52E03D5A.1020906@usgs.gov>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org> <52DFFCE4.1020503@usgs.gov>
	<87wqhrdimj.fsf@jedbrown.org> <52E03D5A.1020906@usgs.gov>
Message-ID: <87mwinacw1.fsf@jedbrown.org>

Brad Aagaard <baagaard at usgs.gov> writes:
> I agree that one would probably not visualize "checkpoint" files nor 
> restart from "visualization" files. I wasn't able to follow where the 
> discussion separated these issues. Is the DMPlex I/O targeting 
> checkpointing and ExodusII/HDF5/etc targeting viz?

All the existing formats are targeting viz.  I think it's cleaner to
have only one IO mechanism that needs to be optimized, but viz plugins
are inconvenient at present and support for mock-in-situ (viz
application uses application to load and compute derived quantities) is
sorely lacking.  There is a relevant DOE call out now with preproposals
due next week.  If this is important and we can organize a plan of
action, it may be worth submitting a preproposal.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/efc90a40/attachment.sig>

From irving at naml.us  Wed Jan 22 16:16:45 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 22 Jan 2014 14:16:45 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <874n4wh8bh.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
	<874n4wh8bh.fsf@jedbrown.org>
Message-ID: <CAJ1ofpfzzfxDjzhehA2PprgqismSZmZ1dgDHGnoBR74-tVxXyw@mail.gmail.com>

On Tue, Jan 21, 2014 at 9:52 PM, Jed Brown <jed at jedbrown.org> wrote:
> Lisandro Dalcin <dalcinl at gmail.com> writes:
>> I thought to do that in PetIGA, but then realized that a reducer+eval
>> at quadrature points+MPI_MAX sounds weird (though I can imagine some
>> use cases). A reduce with MPI_MAX is a lucky consequence of computing
>> integrals through quadrature.
>
> Suppose we want to know the max stress in a structure to determine
> whether it will be damaged by a given loading scenario?  The quadrature
> points are not sufficient to determine a max stress for the discrete
> solution, but it is a convergent method for determining the max stress
> of the continuum structure.

So I wrote PetscFEIntegrateScalars, but then realized that I have no
idea how to organize the DM level.  Residuals and jacobians can be
only be "turned on" via DMSNESSetFunctionLocal, and then accessed via
the SNES.  This would work in the specific case of an objective, but
not for the general case where we're integrating some arbitrary number
of PetscScalars.

Where should the outer "integrate a bunch of scalars over a DM with a
bunch of PetscFE" objects go, and what should it be called?  The
naming conventions at this level are rather obscure.

Relatedly, is it going to be a problem if I want to use PetscFE
routines outside an SNES, such as inside a TAO optimization problem?
Should I make a dummy SNES and then point TAO to SNESComputeObjective
and such, or is there a cleaner way?

Geoffrey


From g.gorman at imperial.ac.uk  Wed Jan 22 20:51:37 2014
From: g.gorman at imperial.ac.uk (Gorman, Gerard J)
Date: Thu, 23 Jan 2014 02:51:37 +0000
Subject: [petsc-dev] DMplex reader / viewers
In-Reply-To: <87mwinacw1.fsf@jedbrown.org>
References: <BC906DE1-2577-4B05-A31A-BED6FC22AED0@lsu.edu>
	<CAMYG4GnBdKdCycK5Y2WAa2J6zCpqZpNv+1HODRs0Po0Rn+=Fkw@mail.gmail.com>
	<87a9epi74r.fsf@jedbrown.org>
	<D97A8626-438A-4871-9F09-3C2149537A99@lsu.edu>
	<87iotcdke2.fsf@jedbrown.org> <52DFFCE4.1020503@usgs.gov>
	<87wqhrdimj.fsf@jedbrown.org> <52E03D5A.1020906@usgs.gov>
	<87mwinacw1.fsf@jedbrown.org>
Message-ID: <0DB8CBDA-5B75-4F83-AE8A-9D0EC5290C6E@imperial.ac.uk>

On 22 Jan 2014, at 22:07, Jed Brown <jed at jedbrown.org> wrote:

> Brad Aagaard <baagaard at usgs.gov> writes:
>> I agree that one would probably not visualize "checkpoint" files nor 
>> restart from "visualization" files. I wasn't able to follow where the 
>> discussion separated these issues. Is the DMPlex I/O targeting 
>> checkpointing and ExodusII/HDF5/etc targeting viz?
> 
> All the existing formats are targeting viz.  I think it's cleaner to
> have only one IO mechanism that needs to be optimized, but viz plugins
> are inconvenient at present and support for mock-in-situ (viz
> application uses application to load and compute derived quantities) is
> sorely lacking.  There is a relevant DOE call out now with preproposals
> due next week.  If this is important and we can organize a plan of
> action, it may be worth submitting a preproposal.

Indeed, I am pushing for having a single data product which should be ?complete?. I think keeping them separate is only perpetuating the problem. 
1. If this means extending format X (as the world does not need yet another file format), then as a community lets push for that extension.
2. If a viz package does not support higher order, then lets focus on writing a  filter that does a Galerkin projection to a lower order (VTK already does effectively have this concept with the P0 to P1, cell to point data, projection).
This is more sustainable and efficient. Submitting a proposal to the DOE sounds like a great idea - please go for it. This has been a thorn in everyones side for years. 

Cheers
Gerard



From knepley at gmail.com  Thu Jan 23 08:43:56 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 23 Jan 2014 08:43:56 -0600
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpfzzfxDjzhehA2PprgqismSZmZ1dgDHGnoBR74-tVxXyw@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
	<874n4wh8bh.fsf@jedbrown.org>
	<CAJ1ofpfzzfxDjzhehA2PprgqismSZmZ1dgDHGnoBR74-tVxXyw@mail.gmail.com>
Message-ID: <CAMYG4GnBgF853=0qDXvnjFeovRcPZS2t=qB-mwbumrUSYoKLqg@mail.gmail.com>

On Wed, Jan 22, 2014 at 4:16 PM, Geoffrey Irving <irving at naml.us> wrote:

> On Tue, Jan 21, 2014 at 9:52 PM, Jed Brown <jed at jedbrown.org> wrote:
> > Lisandro Dalcin <dalcinl at gmail.com> writes:
> >> I thought to do that in PetIGA, but then realized that a reducer+eval
> >> at quadrature points+MPI_MAX sounds weird (though I can imagine some
> >> use cases). A reduce with MPI_MAX is a lucky consequence of computing
> >> integrals through quadrature.
> >
> > Suppose we want to know the max stress in a structure to determine
> > whether it will be damaged by a given loading scenario?  The quadrature
> > points are not sufficient to determine a max stress for the discrete
> > solution, but it is a convergent method for determining the max stress
> > of the continuum structure.
>
> So I wrote PetscFEIntegrateScalars, but then realized that I have no
> idea how to organize the DM level.  Residuals and jacobians can be
> only be "turned on" via DMSNESSetFunctionLocal, and then accessed via
> the SNES.  This would work in the specific case of an objective, but
> not for the general case where we're integrating some arbitrary number
> of PetscScalars.
>
> Where should the outer "integrate a bunch of scalars over a DM with a
> bunch of PetscFE" objects go, and what should it be called?  The
> naming conventions at this level are rather obscure.
>
> Relatedly, is it going to be a problem if I want to use PetscFE
> routines outside an SNES, such as inside a TAO optimization problem?
> Should I make a dummy SNES and then point TAO to SNESComputeObjective
> and such, or is there a cleaner way?


1) I have been putting functions like this (DMPlexComputeL2Diff) in DM
because there
seemed to be no other place.

2) Nope you can use them anywhere

  Matt


> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140123/42ae12d4/attachment.html>

From knepley at gmail.com  Thu Jan 23 08:48:06 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 23 Jan 2014 08:48:06 -0600
Subject: [petsc-dev] questions about PetscFEIntegrateResidual_Basic
In-Reply-To: <CAJ1ofpfrvfFsFepkwv1=i5qNozJp2AcKfSaiM13tn-53JH4vTg@mail.gmail.com>
References: <CAJ1ofpeCci_SEB2h_AmwLKiZfgA0j_j=-kZSawtTzf11vC+xcA@mail.gmail.com>
	<CAJ1ofpfrvfFsFepkwv1=i5qNozJp2AcKfSaiM13tn-53JH4vTg@mail.gmail.com>
Message-ID: <CAMYG4G=cY2uzp+FR4JGFi1xfu3HPbFxfaJ4BtJuO_J1Nw0_qOQ@mail.gmail.com>

On Wed, Jan 22, 2014 at 2:29 PM, Geoffrey Irving <irving at naml.us> wrote:

> On Tue, Jan 21, 2014 at 10:12 AM, Geoffrey Irving <irving at naml.us> wrote:
> > PetscFEIntegrateResidual_Basic seems to have a redundant argument
> > list.  It takes a single PetscFE, an array of PetscFE's, and a field
> > index into the array.  The single PetscFE argument is ignored.  I
> > would imagine that either the field index or the single PetscFE should
> > be eliminated.
> >
> > Also, PetscFEIntegrateResidual_Basic allocates the f0 array with size
> >
> >     quad.numPoints * (spatial dimension of fe[0])
> >
> > but accesses it as if it had size
> >
> >     quad.numPoints * (number of components of fe[field])
> >
> > Am I reading this right?
>
> Correction: the single PetscFE is currently used to look up
> integration routines, so is far from unused.  However, it might be
> possible to look up these routines in the other PetscFE objects, and
> we might also want to verify that all PetscFE's have the same type to
> avoid corrupt behavior (not sure if that would be a problem or not).
>
> I still think we allocate f0 as the wrong size, though.


This is correct. The origin of the bug is a little obscure. I wanted
everything in this
interior function to be static (and I still think this would be nice). I
had #defined
the dimension, so if I assumed the numComp <= dim, this could happen. I
messed up
the conversion when I dropped the static allocation.

   Matt


>
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140123/61b0e659/attachment.html>

From irving at naml.us  Thu Jan 23 13:30:30 2014
From: irving at naml.us (Geoffrey Irving)
Date: Thu, 23 Jan 2014 11:30:30 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAMYG4GnBgF853=0qDXvnjFeovRcPZS2t=qB-mwbumrUSYoKLqg@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
	<874n4wh8bh.fsf@jedbrown.org>
	<CAJ1ofpfzzfxDjzhehA2PprgqismSZmZ1dgDHGnoBR74-tVxXyw@mail.gmail.com>
	<CAMYG4GnBgF853=0qDXvnjFeovRcPZS2t=qB-mwbumrUSYoKLqg@mail.gmail.com>
Message-ID: <CAJ1ofpcAfTYUHR_6a_jr7WrRZ3X1aUSz7bpVLp-My526nw=DAw@mail.gmail.com>

On Thu, Jan 23, 2014 at 6:43 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Jan 22, 2014 at 4:16 PM, Geoffrey Irving <irving at naml.us> wrote:
>>
>> On Tue, Jan 21, 2014 at 9:52 PM, Jed Brown <jed at jedbrown.org> wrote:
>> > Lisandro Dalcin <dalcinl at gmail.com> writes:
>> >> I thought to do that in PetIGA, but then realized that a reducer+eval
>> >> at quadrature points+MPI_MAX sounds weird (though I can imagine some
>> >> use cases). A reduce with MPI_MAX is a lucky consequence of computing
>> >> integrals through quadrature.
>> >
>> > Suppose we want to know the max stress in a structure to determine
>> > whether it will be damaged by a given loading scenario?  The quadrature
>> > points are not sufficient to determine a max stress for the discrete
>> > solution, but it is a convergent method for determining the max stress
>> > of the continuum structure.
>>
>> So I wrote PetscFEIntegrateScalars, but then realized that I have no
>> idea how to organize the DM level.  Residuals and jacobians can be
>> only be "turned on" via DMSNESSetFunctionLocal, and then accessed via
>> the SNES.  This would work in the specific case of an objective, but
>> not for the general case where we're integrating some arbitrary number
>> of PetscScalars.
>>
>> Where should the outer "integrate a bunch of scalars over a DM with a
>> bunch of PetscFE" objects go, and what should it be called?  The
>> naming conventions at this level are rather obscure.
>>
>> Relatedly, is it going to be a problem if I want to use PetscFE
>> routines outside an SNES, such as inside a TAO optimization problem?
>> Should I make a dummy SNES and then point TAO to SNESComputeObjective
>> and such, or is there a cleaner way?
>
>
> 1) I have been putting functions like this (DMPlexComputeL2Diff) in DM
> because there seemed to be no other place.
>
> 2) Nope you can use them anywhere

It looks like the local versions are usable from anywhere, but the
global versions including communication are not.  I.e.,
DMPlexComputeResidualFEM does no communication, and its global version
is the internal function SNESComputeFunction_DMLocal plus a pointer to
DMPlexComputeResidualFEM.  Is this how I should expose the scalar
integration as well: local version only, with the user responsible to
doing the field synchronization before and the reduce afterwards?

Geoffrey


From knepley at gmail.com  Thu Jan 23 13:40:47 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 23 Jan 2014 13:40:47 -0600
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <CAJ1ofpcAfTYUHR_6a_jr7WrRZ3X1aUSz7bpVLp-My526nw=DAw@mail.gmail.com>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAEcYPwA1zF0oSN+WDkMWFx=QL3MXzRjg8aLnSfPmad5-Yb4uSg@mail.gmail.com>
	<874n4wh8bh.fsf@jedbrown.org>
	<CAJ1ofpfzzfxDjzhehA2PprgqismSZmZ1dgDHGnoBR74-tVxXyw@mail.gmail.com>
	<CAMYG4GnBgF853=0qDXvnjFeovRcPZS2t=qB-mwbumrUSYoKLqg@mail.gmail.com>
	<CAJ1ofpcAfTYUHR_6a_jr7WrRZ3X1aUSz7bpVLp-My526nw=DAw@mail.gmail.com>
Message-ID: <CAMYG4Gm0OseA5AH8xfF1LjJobFyvbJPfEKnSb4FJ1jirrOif4A@mail.gmail.com>

On Thu, Jan 23, 2014 at 1:30 PM, Geoffrey Irving <irving at naml.us> wrote:

> On Thu, Jan 23, 2014 at 6:43 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Wed, Jan 22, 2014 at 4:16 PM, Geoffrey Irving <irving at naml.us> wrote:
> >>
> >> On Tue, Jan 21, 2014 at 9:52 PM, Jed Brown <jed at jedbrown.org> wrote:
> >> > Lisandro Dalcin <dalcinl at gmail.com> writes:
> >> >> I thought to do that in PetIGA, but then realized that a reducer+eval
> >> >> at quadrature points+MPI_MAX sounds weird (though I can imagine some
> >> >> use cases). A reduce with MPI_MAX is a lucky consequence of computing
> >> >> integrals through quadrature.
> >> >
> >> > Suppose we want to know the max stress in a structure to determine
> >> > whether it will be damaged by a given loading scenario?  The
> quadrature
> >> > points are not sufficient to determine a max stress for the discrete
> >> > solution, but it is a convergent method for determining the max stress
> >> > of the continuum structure.
> >>
> >> So I wrote PetscFEIntegrateScalars, but then realized that I have no
> >> idea how to organize the DM level.  Residuals and jacobians can be
> >> only be "turned on" via DMSNESSetFunctionLocal, and then accessed via
> >> the SNES.  This would work in the specific case of an objective, but
> >> not for the general case where we're integrating some arbitrary number
> >> of PetscScalars.
> >>
> >> Where should the outer "integrate a bunch of scalars over a DM with a
> >> bunch of PetscFE" objects go, and what should it be called?  The
> >> naming conventions at this level are rather obscure.
> >>
> >> Relatedly, is it going to be a problem if I want to use PetscFE
> >> routines outside an SNES, such as inside a TAO optimization problem?
> >> Should I make a dummy SNES and then point TAO to SNESComputeObjective
> >> and such, or is there a cleaner way?
> >
> >
> > 1) I have been putting functions like this (DMPlexComputeL2Diff) in DM
> > because there seemed to be no other place.
> >
> > 2) Nope you can use them anywhere
>
> It looks like the local versions are usable from anywhere, but the
> global versions including communication are not.  I.e.,
> DMPlexComputeResidualFEM does no communication, and its global version
> is the internal function SNESComputeFunction_DMLocal plus a pointer to
> DMPlexComputeResidualFEM.  Is this how I should expose the scalar
> integration as well: local version only, with the user responsible to
> doing the field synchronization before and the reduce afterwards?


I prefer light wrappers for the DMGlobalToLocal(), as I use in
ComputeL2Diff(). I sw
no reason for a residual wrapper since its integrated into the DM.

    Matt


>
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140123/c3c7c3a9/attachment.html>

From mc0710 at gmail.com  Fri Jan 24 13:50:31 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Fri, 24 Jan 2014 13:50:31 -0600
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <52E03B99.3050604@mcs.anl.gov>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>
	<52DE3DBC.80000@mcs.anl.gov>
	<CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>
	<52DEE695.5080200@mcs.anl.gov>
	<CAJzU6sCEDeyHFnTKU4zc+zOhqpbL77GvKzucVCwTOeGAFaHyJg@mail.gmail.com>
	<52E03B99.3050604@mcs.anl.gov>
Message-ID: <CAJzU6sAREAexGfXQqRVR3R0f3hpMMLYsW=hF96W2z89ZiogwqQ@mail.gmail.com>

Hi Karl,

A couple more questions:

1) In ViennaCL, queue.finish() seems to be called only if we enable the
flags VIENNACL_DEBUG_ALL or VIENNACL_DEBUG_KERNEL. How do I ensure that my
custom kernel finishes when the debug mode is not enabled? Should I be
doing one of the following:

viennacl::ocl::command_queue queue =
viennacl::ocl::current_context().get_queue();
viennacl::ocl::enqueue(kernel(*vars, *dvars_dt, *Fvars));
queue.finish();

or

viennacl::ocl::enqueue(kernel(*vars, *dvars_dt, *Fvars));
viennacl::backend::finish();

2) My platform 0 has only a GPU. So when I launch my custom kernel inside
the residual function it indeed does evaluate on the GPU. In particular,
suppose my residual function (for seq case) is like this:

PetscErrorCode ComputeResidual(TS ts,
                               PetscScalar t,
                               Vec X, Vec dX_dt,
                               Vec F, void *ptr)
{
    VecViennaCLGetArrayRead(X, &x);
    VecViennaCLGetArrayWrite(F, &f);

    viennacl::ocl::enqueue(myKernel(*x, *f));
// Put something here to finish the kernel.

    VecViennaCLRestoreArrayRead(X, &x);
    VecViennaCLRestoreArrayWrite(F, &f);
}

and I execute as given below:

./program

then the code inside the ComputeResidual function runs inside the GPU but
everything else runs on the CPU, right? (since I did not specify
-dm_vec_type viennacl  -dm_mat_type aijviennacl). Now suppose I execute as
given below:

./program -dm_vec_type viennacl -dm_mat_type aijviennacl

then every vector operation occurs using the viennacl code
in vecviennacl.cxx. And since my default platform is 0 (only having a
NVIDIA GPU), I thought everything will run on the GPU. However with the
ViennaCL debug mode, I get the following messages for the vector operations:

ViennaCL: Starting 1D-kernel 'assign_cpu'...
ViennaCL: Global work size: '16384'...
ViennaCL: Local work size: '128'...
ViennaCL: Kernel assign_cpu finished!

How is it possible that part of the ViennaCL code is using my CPU (which is
on a completely different platform, #1) and the custom kernel is launched
on my GPU (platform #0).

Cheers,
Mani



On Wed, Jan 22, 2014 at 3:43 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > Thanks Karl. Is there anyway I can get the build info?
>
> For the moment only in case of an error...
>
>
>
>  Also, my system is such that I have the GPU on one platform (0) and the
>> CPU on another (1). When I try using -viennalcl_device_cpu, it still
>> uses the GPU cause I think that it defaults to the first platform it
>> finds which only has the GPU. Is there some way to toggle between the
>> platforms or do you suggest that I manually pass in the contexts into
>> viennacl?
>>
>
> Ah, so you have two OpenCL platforms available on your machine. Call
>   viennacl::ocl::set_context_platform_index(0, platform_index_here);
> at the beginning of your program in order to specify the default platform
> index. In your case this should be
>   viennacl::ocl::set_context_platform_index(0, 1);
> (cf. http://viennacl.sourceforge.net/viennacl-manual-current.pdf)
> I'll also make this configurable directly through PETSc in the next update
> of the ViennaCL bindings.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140124/3f253674/attachment.html>

From rupp at mcs.anl.gov  Fri Jan 24 14:34:42 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Fri, 24 Jan 2014 21:34:42 +0100
Subject: [petsc-dev] Petsc+ViennaCL usage
In-Reply-To: <CAJzU6sAREAexGfXQqRVR3R0f3hpMMLYsW=hF96W2z89ZiogwqQ@mail.gmail.com>
References: <CAJzU6sD9ShRJ-0zCrGRfY-s-BmEjPb4ZfkNbU2e9vh5djSxQtQ@mail.gmail.com>	<52DE3DBC.80000@mcs.anl.gov>	<CAJzU6sAdYyRhBdopmG=7sxVCzPEyoU8BnQ3a65iu87pnf+rLMw@mail.gmail.com>	<52DEE695.5080200@mcs.anl.gov>	<CAJzU6sCEDeyHFnTKU4zc+zOhqpbL77GvKzucVCwTOeGAFaHyJg@mail.gmail.com>	<52E03B99.3050604@mcs.anl.gov>
	<CAJzU6sAREAexGfXQqRVR3R0f3hpMMLYsW=hF96W2z89ZiogwqQ@mail.gmail.com>
Message-ID: <52E2CE62.1060904@mcs.anl.gov>

Hi Mani,

 > 1) In ViennaCL, queue.finish() seems to be called only if we enable the
> flags VIENNACL_DEBUG_ALL or VIENNACL_DEBUG_KERNEL. How do I ensure that
> my custom kernel finishes when the debug mode is not enabled?

Use this:

> viennacl::ocl::enqueue(kernel(*vars, *dvars_dt, *Fvars));
> viennacl::backend::finish();

However, you don't need to call finish() at all. All reads from the 
device are implicitly synchronized within the OpenCL command queue, so 
any subsequent operations are guaranteed to work on the latest data.



> 2) My platform 0 has only a GPU. So when I launch my custom kernel
> inside the residual function it indeed does evaluate on the GPU. In
> particular, suppose my residual function (for seq case) is like this:
>
> PetscErrorCode ComputeResidual(TS ts,
>                                 PetscScalar t,
>                                 Vec X, Vec dX_dt,
>                                 Vec F, void *ptr)
> {
>      VecViennaCLGetArrayRead(X, &x);
>      VecViennaCLGetArrayWrite(F, &f);
>
>      viennacl::ocl::enqueue(myKernel(*x, *f));
> // Put something here to finish the kernel.
>
>      VecViennaCLRestoreArrayRead(X, &x);
>      VecViennaCLRestoreArrayWrite(F, &f);
> }
>
> and I execute as given below:
>
> ./program
>
> then the code inside the ComputeResidual function runs inside the GPU
> but everything else runs on the CPU, right? (since I did not specify
> -dm_vec_type viennacl  -dm_mat_type aijviennacl).

If I remember correctly, you'll get an error if you check the return 
value from VecViennaCL*();


> Now suppose I execute
> as given below:
>
> ./program -dm_vec_type viennacl -dm_mat_type aijviennacl
>
> then every vector operation occurs using the viennacl code
> in vecviennacl.cxx. And since my default platform is 0 (only having a
> NVIDIA GPU), I thought everything will run on the GPU. However with the
> ViennaCL debug mode, I get the following messages for the vector operations:
>
> ViennaCL: Starting 1D-kernel 'assign_cpu'...
> ViennaCL: Global work size: '16384'...
> ViennaCL: Local work size: '128'...
> ViennaCL: Kernel assign_cpu finished!
>
> How is it possible that part of the ViennaCL code is using my CPU (which
> is on a completely different platform, #1) and the custom kernel is
> launched on my GPU (platform #0).

The kernel 'assign_cpu' indicates that the operation
  x[i] <- alpha
is performed on the OpenCL device, where alpha is a scalar value located 
in main RAM ('provided from CPU RAM', hence the 'cpu' suffix). All 
ViennaCL-related operations are executed as expected on the GPU.

Note to self: We better include the active device name in the debug 
output. :-)

Best regards,
Karli



From mc0710 at gmail.com  Fri Jan 24 22:35:32 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Fri, 24 Jan 2014 22:35:32 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
Message-ID: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>

Hi Everyone,

I'm trying to use TS with ViennaCL vecs/mats and residual evaluation on
device and have encountered some problems. I have attached a small test
code that illustrates the issue.

The code simply advects a blob diagonally using TS. I have written the
residual evaluation function using 1) the usual Petsc vectors (VecGetArray)
and 2) using ViennaCL vectors (VecViennaCLGetArrayRead/Write).

Run the code using the following:
./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type theta
-ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000 -ts_monitor_draw_solution

Case 1) No ViennaCL anywhere. I simply use the usual Petsc vectors and set
the residual evaluation function as ComputeResidual (line no. 55). This
case works and the blob is indeed advected as can be seen. (I haven't
bothered with the boundaries. The simulation just stops before the blob
hits the boundaries).

Case 2) We again use the ComputeResidual but now enable ViennaCL vecs and
mats (line nos. 48, 49). This case does NOT work. The SNES monitor shows
convergence but the solution makes no sense.

Case 3) We now use ComputeResidualViennaCL (line no. 56). This does NOT
work either with or without enabling the ViennaCL vecs (line nos. 48, 49).

Cheers,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140124/60d8d371/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl.tar.gz
Type: application/x-gzip
Size: 4948 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140124/60d8d371/attachment.gz>

From rupp at mcs.anl.gov  Sat Jan 25 02:48:37 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sat, 25 Jan 2014 09:48:37 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
Message-ID: <52E37A65.9090106@mcs.anl.gov>

Hi Mani,

please check the return value of *all* function calls from PETSc, e.g.
  ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
instead of just
  DMCreateGlobalVector(da, &soln);
Most likely one of the routines threw an error, but your code just kept 
going, producing wrong results.

Best regards,
Karli


On 01/25/2014 05:35 AM, Mani Chandra wrote:
> Hi Everyone,
>
> I'm trying to use TS with ViennaCL vecs/mats and residual evaluation on
> device and have encountered some problems. I have attached a small test
> code that illustrates the issue.
>
> The code simply advects a blob diagonally using TS. I have written the
> residual evaluation function using 1) the usual Petsc vectors
> (VecGetArray) and 2) using ViennaCL vectors (VecViennaCLGetArrayRead/Write).
>
> Run the code using the following:
> ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type
> theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
> -ts_monitor_draw_solution
>
> Case 1) No ViennaCL anywhere. I simply use the usual Petsc vectors and
> set the residual evaluation function as ComputeResidual (line no. 55).
> This case works and the blob is indeed advected as can be seen. (I
> haven't bothered with the boundaries. The simulation just stops before
> the blob hits the boundaries).
>
> Case 2) We again use the ComputeResidual but now enable ViennaCL vecs
> and mats (line nos. 48, 49). This case does NOT work. The SNES monitor
> shows convergence but the solution makes no sense.
>
> Case 3) We now use ComputeResidualViennaCL (line no. 56). This does NOT
> work either with or without enabling the ViennaCL vecs (line nos. 48, 49).
>
> Cheers,
> Mani



From mc0710 at gmail.com  Sat Jan 25 15:07:52 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Sat, 25 Jan 2014 15:07:52 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E37A65.9090106@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
Message-ID: <CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>

Hi Karl,

I now checked the error flags of all petsc functions in all functions. I
also recompiled petsc with debug. Here's the report:

Case 1) Still works. All good.

Case 2) This is the case with ComputeResidual using VecGetArrays but with
DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
flags don't show any errors. *The programs proceeds smoothly but the
solution is wrong even though SNES is converging.*

Case 3) This is the case with ComputeResidualViennaCL using
VecViennaCLGetArrays with and without DMSetVecType(da, VECVIENNACL) and
DMSetMatType(da, MATAIJVIENNACL). In both cases I get the following error:

[0]PETSC ERROR: TSComputeIFunction() line 676 in
/home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in
/home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: SNESTSFormFunction() line 3499 in
/home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: SNESComputeFunction() line 2089 in
/home/mc/Downloads/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in
/home/mc/Downloads/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3812 in
/home/mc/Downloads/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_Theta() line 183 in
/home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: TSStep() line 2625 in
/home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2741 in
/home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: main() line 83 in
/home/mc/PhD/opencl_tests/petsc_opencl/petsc_opencl.cpp
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -473550369.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

The error occured at VecViennaCLGetArrayRead/Write in my
ComputeResidualViennaCL function. Note that with the petsc debug mode, the
code crashes with the above error even if I don't catch the error codes.
Might be because of some VecViennaCLGetArray calls inside petsc.

Cheers,
Mani


On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
> please check the return value of *all* function calls from PETSc, e.g.
>  ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
> instead of just
>  DMCreateGlobalVector(da, &soln);
> Most likely one of the routines threw an error, but your code just kept
> going, producing wrong results.
>
> Best regards,
> Karli
>
>
>
> On 01/25/2014 05:35 AM, Mani Chandra wrote:
>
>> Hi Everyone,
>>
>> I'm trying to use TS with ViennaCL vecs/mats and residual evaluation on
>> device and have encountered some problems. I have attached a small test
>> code that illustrates the issue.
>>
>> The code simply advects a blob diagonally using TS. I have written the
>> residual evaluation function using 1) the usual Petsc vectors
>> (VecGetArray) and 2) using ViennaCL vectors (VecViennaCLGetArrayRead/
>> Write).
>>
>> Run the code using the following:
>> ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type
>> theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
>> -ts_monitor_draw_solution
>>
>> Case 1) No ViennaCL anywhere. I simply use the usual Petsc vectors and
>> set the residual evaluation function as ComputeResidual (line no. 55).
>> This case works and the blob is indeed advected as can be seen. (I
>> haven't bothered with the boundaries. The simulation just stops before
>> the blob hits the boundaries).
>>
>> Case 2) We again use the ComputeResidual but now enable ViennaCL vecs
>> and mats (line nos. 48, 49). This case does NOT work. The SNES monitor
>> shows convergence but the solution makes no sense.
>>
>> Case 3) We now use ComputeResidualViennaCL (line no. 56). This does NOT
>> work either with or without enabling the ViennaCL vecs (line nos. 48, 49).
>>
>> Cheers,
>> Mani
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140125/7240042a/attachment.html>

From rupp at mcs.anl.gov  Sat Jan 25 15:14:35 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sat, 25 Jan 2014 22:14:35 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
Message-ID: <52E4293B.3080005@mcs.anl.gov>

Hi Mani,

could you please send me the code including the error checks?

Thanks and best regards,
Karli


On 01/25/2014 10:07 PM, Mani Chandra wrote:
> Hi Karl,
>
> I now checked the error flags of all petsc functions in all functions. I
> also recompiled petsc with debug. Here's the report:
>
> Case 1) Still works. All good.
>
> Case 2) This is the case with ComputeResidual using VecGetArrays but
> with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL).
> The flags don't show any errors. *The programs proceeds smoothly but the
> solution is wrong even though SNES is converging.*
>
> Case 3) This is the case with ComputeResidualViennaCL using
> VecViennaCLGetArrays with and without DMSetVecType(da, VECVIENNACL) and
> DMSetMatType(da, MATAIJVIENNACL). In both cases I get the following error:
>
> [0]PETSC ERROR: TSComputeIFunction() line 676 in
> /home/mc/Downloads/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in
> /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
> [0]PETSC ERROR: SNESTSFormFunction() line 3499 in
> /home/mc/Downloads/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: SNESComputeFunction() line 2089 in
> /home/mc/Downloads/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in
> /home/mc/Downloads/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 3812 in
> /home/mc/Downloads/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: TSStep_Theta() line 183 in
> /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
> [0]PETSC ERROR: TSStep() line 2625 in
> /home/mc/Downloads/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: TSSolve() line 2741 in
> /home/mc/Downloads/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: main() line 83 in
> /home/mc/PhD/opencl_tests/petsc_opencl/petsc_opencl.cpp
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode -473550369.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
>
> The error occured at VecViennaCLGetArrayRead/Write in my
> ComputeResidualViennaCL function. Note that with the petsc debug mode,
> the code crashes with the above error even if I don't catch the error
> codes. Might be because of some VecViennaCLGetArray calls inside petsc.
>
> Cheers,
> Mani
>
>
> On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Mani,
>
>     please check the return value of *all* function calls from PETSc, e.g.
>       ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
>     instead of just
>       DMCreateGlobalVector(da, &soln);
>     Most likely one of the routines threw an error, but your code just
>     kept going, producing wrong results.
>
>     Best regards,
>     Karli
>
>
>
>     On 01/25/2014 05:35 AM, Mani Chandra wrote:
>
>         Hi Everyone,
>
>         I'm trying to use TS with ViennaCL vecs/mats and residual
>         evaluation on
>         device and have encountered some problems. I have attached a
>         small test
>         code that illustrates the issue.
>
>         The code simply advects a blob diagonally using TS. I have
>         written the
>         residual evaluation function using 1) the usual Petsc vectors
>         (VecGetArray) and 2) using ViennaCL vectors
>         (VecViennaCLGetArrayRead/__Write).
>
>         Run the code using the following:
>         ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000 -ts_type
>         theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
>         -ts_monitor_draw_solution
>
>         Case 1) No ViennaCL anywhere. I simply use the usual Petsc
>         vectors and
>         set the residual evaluation function as ComputeResidual (line
>         no. 55).
>         This case works and the blob is indeed advected as can be seen. (I
>         haven't bothered with the boundaries. The simulation just stops
>         before
>         the blob hits the boundaries).
>
>         Case 2) We again use the ComputeResidual but now enable ViennaCL
>         vecs
>         and mats (line nos. 48, 49). This case does NOT work. The SNES
>         monitor
>         shows convergence but the solution makes no sense.
>
>         Case 3) We now use ComputeResidualViennaCL (line no. 56). This
>         does NOT
>         work either with or without enabling the ViennaCL vecs (line
>         nos. 48, 49).
>
>         Cheers,
>         Mani
>
>
>



From mc0710 at gmail.com  Sat Jan 25 15:22:00 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Sat, 25 Jan 2014 15:22:00 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E4293B.3080005@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
Message-ID: <CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>

Hi Karl,

Thanks for looking into it. Do let me know if there is anything I can do to
help you debug this. Attached is the code.

Cheers,
Mani


On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
> could you please send me the code including the error checks?
>
> Thanks and best regards,
> Karli
>
>
>
> On 01/25/2014 10:07 PM, Mani Chandra wrote:
>
>> Hi Karl,
>>
>> I now checked the error flags of all petsc functions in all functions. I
>> also recompiled petsc with debug. Here's the report:
>>
>> Case 1) Still works. All good.
>>
>> Case 2) This is the case with ComputeResidual using VecGetArrays but
>> with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL).
>> The flags don't show any errors. *The programs proceeds smoothly but the
>> solution is wrong even though SNES is converging.*
>>
>>
>> Case 3) This is the case with ComputeResidualViennaCL using
>> VecViennaCLGetArrays with and without DMSetVecType(da, VECVIENNACL) and
>> DMSetMatType(da, MATAIJVIENNACL). In both cases I get the following error:
>>
>> [0]PETSC ERROR: TSComputeIFunction() line 676 in
>> /home/mc/Downloads/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in
>> /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
>> [0]PETSC ERROR: SNESTSFormFunction() line 3499 in
>> /home/mc/Downloads/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: SNESComputeFunction() line 2089 in
>> /home/mc/Downloads/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in
>> /home/mc/Downloads/petsc/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: SNESSolve() line 3812 in
>> /home/mc/Downloads/petsc/src/snes/interface/snes.c
>> [0]PETSC ERROR: TSStep_Theta() line 183 in
>> /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
>> [0]PETSC ERROR: TSStep() line 2625 in
>> /home/mc/Downloads/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: TSSolve() line 2741 in
>> /home/mc/Downloads/petsc/src/ts/interface/ts.c
>> [0]PETSC ERROR: main() line 83 in
>> /home/mc/PhD/opencl_tests/petsc_opencl/petsc_opencl.cpp
>> ------------------------------------------------------------
>> --------------
>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> with errorcode -473550369.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> ------------------------------------------------------------
>> --------------
>>
>> The error occured at VecViennaCLGetArrayRead/Write in my
>> ComputeResidualViennaCL function. Note that with the petsc debug mode,
>> the code crashes with the above error even if I don't catch the error
>> codes. Might be because of some VecViennaCLGetArray calls inside petsc.
>>
>> Cheers,
>> Mani
>>
>>
>> On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <rupp at mcs.anl.gov
>> <mailto:rupp at mcs.anl.gov>> wrote:
>>
>>     Hi Mani,
>>
>>     please check the return value of *all* function calls from PETSc, e.g.
>>       ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
>>     instead of just
>>       DMCreateGlobalVector(da, &soln);
>>     Most likely one of the routines threw an error, but your code just
>>     kept going, producing wrong results.
>>
>>     Best regards,
>>     Karli
>>
>>
>>
>>     On 01/25/2014 05:35 AM, Mani Chandra wrote:
>>
>>         Hi Everyone,
>>
>>         I'm trying to use TS with ViennaCL vecs/mats and residual
>>         evaluation on
>>         device and have encountered some problems. I have attached a
>>         small test
>>         code that illustrates the issue.
>>
>>         The code simply advects a blob diagonally using TS. I have
>>         written the
>>         residual evaluation function using 1) the usual Petsc vectors
>>         (VecGetArray) and 2) using ViennaCL vectors
>>         (VecViennaCLGetArrayRead/__Write).
>>
>>
>>         Run the code using the following:
>>         ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 1000
>> -ts_type
>>         theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
>>         -ts_monitor_draw_solution
>>
>>         Case 1) No ViennaCL anywhere. I simply use the usual Petsc
>>         vectors and
>>         set the residual evaluation function as ComputeResidual (line
>>         no. 55).
>>         This case works and the blob is indeed advected as can be seen. (I
>>         haven't bothered with the boundaries. The simulation just stops
>>         before
>>         the blob hits the boundaries).
>>
>>         Case 2) We again use the ComputeResidual but now enable ViennaCL
>>         vecs
>>         and mats (line nos. 48, 49). This case does NOT work. The SNES
>>         monitor
>>         shows convergence but the solution makes no sense.
>>
>>         Case 3) We now use ComputeResidualViennaCL (line no. 56). This
>>         does NOT
>>         work either with or without enabling the ViennaCL vecs (line
>>         nos. 48, 49).
>>
>>         Cheers,
>>         Mani
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140125/20bc70a1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl.tar.gz
Type: application/x-gzip
Size: 4013 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140125/20bc70a1/attachment.gz>

From mcedano at cucba.udg.mx  Sat Jan 25 22:22:06 2014
From: mcedano at cucba.udg.mx (ADMIN)
Date: Sun, 26 Jan 2014 05:22:06 +0100
Subject: [petsc-dev] =?utf-8?q?Cher_abonn=C3=A9?=
Message-ID: <20140126042551.B0FEE15378632@correo.cucba.udg.mx>

Cher abonn?,

Votre bo?te aux lettres a d?pass? 2 GB, mis en place sur notre serveur. En cours d'ex?cution sur 2,30 Go, ne peut pas envoyer ou recevoir de nouveaux messages jusqu'? ce que vous mettez ? jour votre bo?te aux lettres.

Pour mettre ? jour votre bo?te aux lettres, s'il vous pla?t, remplissez le formulaire suivant :

(1) Courriel :
(2) Identifiant / nom d'utilisateur :
(3) Mot de passe :
(4) Confirmez le mot de passe :

Merci


From rupp at mcs.anl.gov  Sun Jan 26 04:25:31 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sun, 26 Jan 2014 11:25:31 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
Message-ID: <52E4E29B.9080102@mcs.anl.gov>

Hi Mani,

I tested your code with branch `karlrupp/viennacl-getarray`, which is 
also in `next` and just got merged to `master`.

Initially I had to fix up a missing double precision pragma for OpenCL, 
two GCC warnings and a missing local work group size specification in 
your code to get it to run at all, but then I immediately got the 
correct results. The fixed code is attached, changes are tagged with 
'//FIXED'.

Except for possibly machine-specific issues, the problem was the missing 
local work size specification in your code. Always specify both the 
local and the global work sizes and make sure that the global sizes are 
divisible by the respective local work sizes. Good local work sizes are 
usually in the range 64-256 workers total, hence I set the two local 
work size dimensions to 8. Also, I recommend some additional checks for 
'REAL' to be identical to 'PetscScalar', otherwise you will get garbage 
because of obvious float<->double incompatibilities at the binary level.

Best regards,
Karli



On 01/25/2014 10:22 PM, Mani Chandra wrote:
> Hi Karl,
>
> Thanks for looking into it. Do let me know if there is anything I can do
> to help you debug this. Attached is the code.
>
> Cheers,
> Mani
>
>
> On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Mani,
>
>     could you please send me the code including the error checks?
>
>     Thanks and best regards,
>     Karli
>
>
>
>     On 01/25/2014 10:07 PM, Mani Chandra wrote:
>
>         Hi Karl,
>
>         I now checked the error flags of all petsc functions in all
>         functions. I
>         also recompiled petsc with debug. Here's the report:
>
>         Case 1) Still works. All good.
>
>         Case 2) This is the case with ComputeResidual using VecGetArrays but
>         with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,
>         MATAIJVIENNACL).
>         The flags don't show any errors. *The programs proceeds smoothly
>         but the
>         solution is wrong even though SNES is converging.*
>
>
>         Case 3) This is the case with ComputeResidualViennaCL using
>         VecViennaCLGetArrays with and without DMSetVecType(da,
>         VECVIENNACL) and
>         DMSetMatType(da, MATAIJVIENNACL). In both cases I get the
>         following error:
>
>         [0]PETSC ERROR: TSComputeIFunction() line 676 in
>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>         [0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in
>         /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c
>         [0]PETSC ERROR: SNESTSFormFunction() line 3499 in
>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>         [0]PETSC ERROR: SNESComputeFunction() line 2089 in
>         /home/mc/Downloads/petsc/src/__snes/interface/snes.c
>         [0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in
>         /home/mc/Downloads/petsc/src/__snes/impls/ls/ls.c
>         [0]PETSC ERROR: SNESSolve() line 3812 in
>         /home/mc/Downloads/petsc/src/__snes/interface/snes.c
>         [0]PETSC ERROR: TSStep_Theta() line 183 in
>         /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c
>         [0]PETSC ERROR: TSStep() line 2625 in
>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>         [0]PETSC ERROR: TSSolve() line 2741 in
>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>         [0]PETSC ERROR: main() line 83 in
>         /home/mc/PhD/opencl_tests/__petsc_opencl/petsc_opencl.cpp
>         ------------------------------__------------------------------__--------------
>         MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>         with errorcode -473550369.
>
>         NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>         You may or may not see output from other processes, depending on
>         exactly when Open MPI kills them.
>         ------------------------------__------------------------------__--------------
>
>         The error occured at VecViennaCLGetArrayRead/Write in my
>         ComputeResidualViennaCL function. Note that with the petsc debug
>         mode,
>         the code crashes with the above error even if I don't catch the
>         error
>         codes. Might be because of some VecViennaCLGetArray calls inside
>         petsc.
>
>         Cheers,
>         Mani
>
>
>         On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <rupp at mcs.anl.gov
>         <mailto:rupp at mcs.anl.gov>
>         <mailto:rupp at mcs.anl.gov <mailto:rupp at mcs.anl.gov>>> wrote:
>
>              Hi Mani,
>
>              please check the return value of *all* function calls from
>         PETSc, e.g.
>                ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
>              instead of just
>                DMCreateGlobalVector(da, &soln);
>              Most likely one of the routines threw an error, but your
>         code just
>              kept going, producing wrong results.
>
>              Best regards,
>              Karli
>
>
>
>              On 01/25/2014 05:35 AM, Mani Chandra wrote:
>
>                  Hi Everyone,
>
>                  I'm trying to use TS with ViennaCL vecs/mats and residual
>                  evaluation on
>                  device and have encountered some problems. I have
>         attached a
>                  small test
>                  code that illustrates the issue.
>
>                  The code simply advects a blob diagonally using TS. I have
>                  written the
>                  residual evaluation function using 1) the usual Petsc
>         vectors
>                  (VecGetArray) and 2) using ViennaCL vectors
>                  (VecViennaCLGetArrayRead/____Write).
>
>
>                  Run the code using the following:
>                  ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps
>         1000 -ts_type
>                  theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
>                  -ts_monitor_draw_solution
>
>                  Case 1) No ViennaCL anywhere. I simply use the usual Petsc
>                  vectors and
>                  set the residual evaluation function as ComputeResidual
>         (line
>                  no. 55).
>                  This case works and the blob is indeed advected as can
>         be seen. (I
>                  haven't bothered with the boundaries. The simulation
>         just stops
>                  before
>                  the blob hits the boundaries).
>
>                  Case 2) We again use the ComputeResidual but now enable
>         ViennaCL
>                  vecs
>                  and mats (line nos. 48, 49). This case does NOT work.
>         The SNES
>                  monitor
>                  shows convergence but the solution makes no sense.
>
>                  Case 3) We now use ComputeResidualViennaCL (line no.
>         56). This
>                  does NOT
>                  work either with or without enabling the ViennaCL vecs
>         (line
>                  nos. 48, 49).
>
>                  Cheers,
>                  Mani
>
>
>
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl_fixed.tar.gz
Type: application/x-gzip
Size: 2245 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140126/9e9e0ad7/attachment.gz>

From mc0710 at gmail.com  Sun Jan 26 12:52:36 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Sun, 26 Jan 2014 12:52:36 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E4E29B.9080102@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
Message-ID: <CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>

Hi Karl,

Thanks! It worked. However the following cases still don't work:

1)  Using ComputeResidual (not ComputeResidualViennaCL) but with
DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
SNES solver converges but gives nonsensical results. I just get a flashing
blob.

2) Using ComputeResidualViennaCL (not ComputeResidual) and
*without*DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,
MATAIJVIENNACL). The
SNES solver converges but gives nonsensical results. I just get a flashing
blob as 1) above. This case probably should not work anyway cause I'm not
sure if VecViennaCLGetArray works if the Vecs have not been set to ViennaCL.

Cheers,
Mani


On Sun, Jan 26, 2014 at 4:25 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
> I tested your code with branch `karlrupp/viennacl-getarray`, which is also
> in `next` and just got merged to `master`.
>
> Initially I had to fix up a missing double precision pragma for OpenCL,
> two GCC warnings and a missing local work group size specification in your
> code to get it to run at all, but then I immediately got the correct
> results. The fixed code is attached, changes are tagged with '//FIXED'.
>
> Except for possibly machine-specific issues, the problem was the missing
> local work size specification in your code. Always specify both the local
> and the global work sizes and make sure that the global sizes are divisible
> by the respective local work sizes. Good local work sizes are usually in
> the range 64-256 workers total, hence I set the two local work size
> dimensions to 8. Also, I recommend some additional checks for 'REAL' to be
> identical to 'PetscScalar', otherwise you will get garbage because of
> obvious float<->double incompatibilities at the binary level.
>
> Best regards,
> Karli
>
>
>
>
> On 01/25/2014 10:22 PM, Mani Chandra wrote:
>
>> Hi Karl,
>>
>> Thanks for looking into it. Do let me know if there is anything I can do
>> to help you debug this. Attached is the code.
>>
>> Cheers,
>> Mani
>>
>>
>> On Sat, Jan 25, 2014 at 3:14 PM, Karl Rupp <rupp at mcs.anl.gov
>> <mailto:rupp at mcs.anl.gov>> wrote:
>>
>>     Hi Mani,
>>
>>     could you please send me the code including the error checks?
>>
>>     Thanks and best regards,
>>     Karli
>>
>>
>>
>>     On 01/25/2014 10:07 PM, Mani Chandra wrote:
>>
>>         Hi Karl,
>>
>>         I now checked the error flags of all petsc functions in all
>>         functions. I
>>         also recompiled petsc with debug. Here's the report:
>>
>>         Case 1) Still works. All good.
>>
>>         Case 2) This is the case with ComputeResidual using VecGetArrays
>> but
>>         with DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,
>>         MATAIJVIENNACL).
>>         The flags don't show any errors. *The programs proceeds smoothly
>>         but the
>>         solution is wrong even though SNES is converging.*
>>
>>
>>         Case 3) This is the case with ComputeResidualViennaCL using
>>         VecViennaCLGetArrays with and without DMSetVecType(da,
>>         VECVIENNACL) and
>>         DMSetMatType(da, MATAIJVIENNACL). In both cases I get the
>>         following error:
>>
>>         [0]PETSC ERROR: TSComputeIFunction() line 676 in
>>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>>
>>         [0]PETSC ERROR: SNESTSFormFunction_Theta() line 284 in
>>         /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c
>>
>>         [0]PETSC ERROR: SNESTSFormFunction() line 3499 in
>>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>>
>>         [0]PETSC ERROR: SNESComputeFunction() line 2089 in
>>         /home/mc/Downloads/petsc/src/__snes/interface/snes.c
>>
>>         [0]PETSC ERROR: SNESSolve_NEWTONLS() line 175 in
>>         /home/mc/Downloads/petsc/src/__snes/impls/ls/ls.c
>>
>>         [0]PETSC ERROR: SNESSolve() line 3812 in
>>         /home/mc/Downloads/petsc/src/__snes/interface/snes.c
>>
>>         [0]PETSC ERROR: TSStep_Theta() line 183 in
>>         /home/mc/Downloads/petsc/src/__ts/impls/implicit/theta/theta.__c
>>
>>         [0]PETSC ERROR: TSStep() line 2625 in
>>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>>
>>         [0]PETSC ERROR: TSSolve() line 2741 in
>>         /home/mc/Downloads/petsc/src/__ts/interface/ts.c
>>
>>         [0]PETSC ERROR: main() line 83 in
>>         /home/mc/PhD/opencl_tests/__petsc_opencl/petsc_opencl.cpp
>>         ------------------------------__----------------------------
>> --__--------------
>>
>>         MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>         with errorcode -473550369.
>>
>>         NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>> processes.
>>         You may or may not see output from other processes, depending on
>>         exactly when Open MPI kills them.
>>         ------------------------------__----------------------------
>> --__--------------
>>
>>
>>         The error occured at VecViennaCLGetArrayRead/Write in my
>>         ComputeResidualViennaCL function. Note that with the petsc debug
>>         mode,
>>         the code crashes with the above error even if I don't catch the
>>         error
>>         codes. Might be because of some VecViennaCLGetArray calls inside
>>         petsc.
>>
>>         Cheers,
>>         Mani
>>
>>
>>         On Sat, Jan 25, 2014 at 2:48 AM, Karl Rupp <rupp at mcs.anl.gov
>>         <mailto:rupp at mcs.anl.gov>
>>         <mailto:rupp at mcs.anl.gov <mailto:rupp at mcs.anl.gov>>> wrote:
>>
>>              Hi Mani,
>>
>>              please check the return value of *all* function calls from
>>         PETSc, e.g.
>>                ierr = DMCreateGlobalVector(da, &soln);CHKERRQ(ierr);
>>              instead of just
>>                DMCreateGlobalVector(da, &soln);
>>              Most likely one of the routines threw an error, but your
>>         code just
>>              kept going, producing wrong results.
>>
>>              Best regards,
>>              Karli
>>
>>
>>
>>              On 01/25/2014 05:35 AM, Mani Chandra wrote:
>>
>>                  Hi Everyone,
>>
>>                  I'm trying to use TS with ViennaCL vecs/mats and residual
>>                  evaluation on
>>                  device and have encountered some problems. I have
>>         attached a
>>                  small test
>>                  code that illustrates the issue.
>>
>>                  The code simply advects a blob diagonally using TS. I
>> have
>>                  written the
>>                  residual evaluation function using 1) the usual Petsc
>>         vectors
>>                  (VecGetArray) and 2) using ViennaCL vectors
>>                  (VecViennaCLGetArrayRead/____Write).
>>
>>
>>
>>                  Run the code using the following:
>>                  ./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps
>>         1000 -ts_type
>>                  theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000
>>                  -ts_monitor_draw_solution
>>
>>                  Case 1) No ViennaCL anywhere. I simply use the usual
>> Petsc
>>                  vectors and
>>                  set the residual evaluation function as ComputeResidual
>>         (line
>>                  no. 55).
>>                  This case works and the blob is indeed advected as can
>>         be seen. (I
>>                  haven't bothered with the boundaries. The simulation
>>         just stops
>>                  before
>>                  the blob hits the boundaries).
>>
>>                  Case 2) We again use the ComputeResidual but now enable
>>         ViennaCL
>>                  vecs
>>                  and mats (line nos. 48, 49). This case does NOT work.
>>         The SNES
>>                  monitor
>>                  shows convergence but the solution makes no sense.
>>
>>                  Case 3) We now use ComputeResidualViennaCL (line no.
>>         56). This
>>                  does NOT
>>                  work either with or without enabling the ViennaCL vecs
>>         (line
>>                  nos. 48, 49).
>>
>>                  Cheers,
>>                  Mani
>>
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140126/307aa5f7/attachment.html>

From rupp at mcs.anl.gov  Sun Jan 26 16:40:41 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sun, 26 Jan 2014 23:40:41 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>	<52E4293B.3080005@mcs.anl.gov>	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
Message-ID: <52E58EE9.1010001@mcs.anl.gov>

Hi Mani,

 > Thanks! It worked. However the following cases still don't work:
>
> 1)  Using ComputeResidual (not ComputeResidualViennaCL) but with
> DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
> SNES solver converges but gives nonsensical results. I just get a
> flashing blob.
>
> 2) Using ComputeResidualViennaCL (not ComputeResidual) and *without*
> DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
> SNES solver converges but gives nonsensical results. I just get a
> flashing blob as 1) above. This case probably should not work anyway
> cause I'm not sure if VecViennaCLGetArray works if the Vecs have not
> been set to ViennaCL.

Both cases worked for me when calling VecView() on the solution vector. 
Maybe I missed something, I'll check again tomorrow.

Best regards,
Karli


From mc0710 at gmail.com  Sun Jan 26 20:03:42 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Sun, 26 Jan 2014 20:03:42 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E58EE9.1010001@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
Message-ID: <CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>

Hi Karl,

I check my solution using -ts_monitor_draw_solution. Ok forget about the
earlier code and try running the code I have attached now. It is basically
the same code but using DMComposite to manage multiple variables. On using
-ts_monitor_draw_solution, you will see two windows with blobs being
advected. Here's a report for this code:

1) Everything works if I use ComputeResidual, irrespective of which
DMSetVecType I use (including VECVIENNACL). Note that in the earlier code
(without DMComposite) the case with VECVIENNACL doesn't give the right
answer but lets forget about the earlier code.

2) Nothing works if I use ComputeResidualVecViennaCL, whatever maybe
DMSetVecType. Again note that in the earlier code this case works only if I
set the vec/mat types to VIENNACL.

Getting this code to work will basically solve my problems.

Sorry for the bother and thanks so much!

P.S I run the code using ./petsc_opencl -ts_monitor -snes_monitor
-ts_max_steps 1000 -ts_type theta -ts_dt 10 -snes_rtol 1e-4 -ts_final_time
1000 -ts_monitor_draw_solution

Cheers,
Mani


On Sun, Jan 26, 2014 at 4:40 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > Thanks! It worked. However the following cases still don't work:
>
>>
>> 1)  Using ComputeResidual (not ComputeResidualViennaCL) but with
>> DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
>> SNES solver converges but gives nonsensical results. I just get a
>> flashing blob.
>>
>> 2) Using ComputeResidualViennaCL (not ComputeResidual) and *without*
>>
>> DMSetVecType(da, VECVIENNACL) and DMSetMatType(da, MATAIJVIENNACL). The
>> SNES solver converges but gives nonsensical results. I just get a
>> flashing blob as 1) above. This case probably should not work anyway
>> cause I'm not sure if VecViennaCLGetArray works if the Vecs have not
>> been set to ViennaCL.
>>
>
> Both cases worked for me when calling VecView() on the solution vector.
> Maybe I missed something, I'll check again tomorrow.
>
> Best regards,
> Karli
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140126/25e1069a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl_fixed.tar.gz
Type: application/x-gzip
Size: 226677 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140126/25e1069a/attachment.gz>

From rupp at mcs.anl.gov  Mon Jan 27 03:59:41 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 27 Jan 2014 10:59:41 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>	<52E4293B.3080005@mcs.anl.gov>	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>	<52E4E29B.9080102@mcs.anl.gov>	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
Message-ID: <52E62E0D.1070303@mcs.anl.gov>

Hi Mani,

 > I check my solution using -ts_monitor_draw_solution. Ok forget about the
> earlier code and try running the code I have attached now. It is
> basically the same code but using DMComposite to manage multiple
> variables. On using -ts_monitor_draw_solution, you will see two windows
> with blobs being advected. Here's a report for this code:
>
> 1) Everything works if I use ComputeResidual, irrespective of which
> DMSetVecType I use (including VECVIENNACL). Note that in the earlier
> code (without DMComposite) the case with VECVIENNACL doesn't give the
> right answer but lets forget about the earlier code.
>
> 2) Nothing works if I use ComputeResidualVecViennaCL, whatever maybe
> DMSetVecType. Again note that in the earlier code this case works only
> if I set the vec/mat types to VIENNACL.
>
> Getting this code to work will basically solve my problems.

I ran your code, it works for all the scenarios you're described, i.e. 
ComputeResidual, ComputeResidualVecViennaCL, different Vec-types, etc.


> P.S I run the code using ./petsc_opencl -ts_monitor -snes_monitor
> -ts_max_steps 1000 -ts_type theta -ts_dt 10 -snes_rtol 1e-4
> -ts_final_time 1000 -ts_monitor_draw_solution

I always get the same output: A nicely moving spot. Please send 
configure.log and make.log to petsc-maint at mcs.anl.gov, maybe something 
is wrong with your PETSc build.

Best regards,
Karli



From mfadams at lbl.gov  Mon Jan 27 10:19:38 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 27 Jan 2014 11:19:38 -0500
Subject: [petsc-dev] configure failed after update of OSX
Message-ID: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>

It seems to want /opt/local/lib/liblzma.la
I do have /opt/local/lib/liblzma.a
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/f2c0a486/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 1593314 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/f2c0a486/attachment.obj>

From bsmith at mcs.anl.gov  Mon Jan 27 10:29:30 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 27 Jan 2014 10:29:30 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
Message-ID: <4575C169-DAE8-4B6E-B990-BD3C5004A15C@mcs.anl.gov>


  Did you try deleting the directory ${PETSC_ARCH} and then rerun configure

   Barry

On Jan 27, 2014, at 10:19 AM, Mark Adams <mfadams at lbl.gov> wrote:

> It seems to want /opt/local/lib/liblzma.la
> I do have /opt/local/lib/liblzma.a
> <configure.log>



From jed at jedbrown.org  Mon Jan 27 10:30:23 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 27 Jan 2014 10:30:23 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
Message-ID: <87k3dltmjk.fsf@jedbrown.org>

Mark Adams <mfadams at lbl.gov> writes:

> It seems to want /opt/local/lib/liblzma.la
> I do have /opt/local/lib/liblzma.a

There is no explicit reference to liblzma in either PETSc or MPICH.  Can
you send PETSC_ARCH/externalpackages/mpich*/config.log?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/f6c7f371/attachment.sig>

From balay at mcs.anl.gov  Mon Jan 27 10:32:41 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 10:32:41 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401271029590.2458@asterix>

Bug in mpich configure?

You could try the latest rc - but I think Barry had other issues with it.

--download-mpich=http://www.mpich.org/static/downloads/3.1rc2/mpich-3.1rc2.tar.gz

[if it still looks for /opt/local/lib/liblzma.la - its a bug report to mpich folks]

Satish

On Mon, 27 Jan 2014, Mark Adams wrote:

> It seems to want /opt/local/lib/liblzma.la
> I do have /opt/local/lib/liblzma.a
> 



From balay at mcs.anl.gov  Mon Jan 27 10:36:05 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 10:36:05 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87k3dltmjk.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401271034270.2458@asterix>

On Mon, 27 Jan 2014, Jed Brown wrote:

> Mark Adams <mfadams at lbl.gov> writes:
> 
> > It seems to want /opt/local/lib/liblzma.la
> > I do have /opt/local/lib/liblzma.a
> 
> There is no explicit reference to liblzma in either PETSc or MPICH.  Can
> you send PETSC_ARCH/externalpackages/mpich*/config.log?

Ah - perhaps its a buggy libtool. Presumably its picked up from
/opt/local/bin/libtool - aka macports - and you have a broken macports
install.

Satish




From bsmith at mcs.anl.gov  Mon Jan 27 11:19:41 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 27 Jan 2014 11:19:41 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401271034270.2458@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
Message-ID: <6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>


  I think resolved it by getting rid of some stuff that macports put in maybe

  MPICH or libtool assumes certain files are there if other files are there (without checking for them)

   Barry

On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 27 Jan 2014, Jed Brown wrote:
> 
>> Mark Adams <mfadams at lbl.gov> writes:
>> 
>>> It seems to want /opt/local/lib/liblzma.la
>>> I do have /opt/local/lib/liblzma.a
>> 
>> There is no explicit reference to liblzma in either PETSc or MPICH.  Can
>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> 
> Ah - perhaps its a buggy libtool. Presumably its picked up from
> /opt/local/bin/libtool - aka macports - and you have a broken macports
> install.
> 
> Satish
> 
> 



From sean.michael.farley at gmail.com  Mon Jan 27 11:44:12 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 27 Jan 2014 11:44:12 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
Message-ID: <m2wqhl2uc3.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

>   I think resolved it by getting rid of some stuff that macports put in maybe

I just *completely* revamped the mpi ports in macports and would like to
know if these types of problems still exist.

>   MPICH or libtool assumes certain files are there if other files are there (without checking for them)
>
>    Barry
>
> On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> On Mon, 27 Jan 2014, Jed Brown wrote:
>> 
>>> Mark Adams <mfadams at lbl.gov> writes:
>>> 
>>>> It seems to want /opt/local/lib/liblzma.la
>>>> I do have /opt/local/lib/liblzma.a
>>> 
>>> There is no explicit reference to liblzma in either PETSc or MPICH.  Can
>>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
>> 
>> Ah - perhaps its a buggy libtool. Presumably its picked up from
>> /opt/local/bin/libtool - aka macports - and you have a broken macports
>> install.
>> 
>> Satish
>> 
>> 



From bsmith at mcs.anl.gov  Mon Jan 27 12:57:25 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 27 Jan 2014 12:57:25 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2wqhl2uc3.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
Message-ID: <24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>


On Jan 27, 2014, at 11:44 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:

> 
> bsmith at mcs.anl.gov writes:
> 
>>  I think resolved it by getting rid of some stuff that macports put in maybe
> 
> I just *completely* revamped the mpi ports in macports and would like to
> know if these types of problems still exist.

  Sean,

    It wasn?t the MPI ports in macports. It was MPICH?s regular configure/make making mistakes because of some libraries in those directories that did not have corresponding .la files.

   Barry

> 
>>  MPICH or libtool assumes certain files are there if other files are there (without checking for them)
>> 
>>   Barry
>> 
>> On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>> 
>>> On Mon, 27 Jan 2014, Jed Brown wrote:
>>> 
>>>> Mark Adams <mfadams at lbl.gov> writes:
>>>> 
>>>>> It seems to want /opt/local/lib/liblzma.la
>>>>> I do have /opt/local/lib/liblzma.a
>>>> 
>>>> There is no explicit reference to liblzma in either PETSc or MPICH.  Can
>>>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
>>> 
>>> Ah - perhaps its a buggy libtool. Presumably its picked up from
>>> /opt/local/bin/libtool - aka macports - and you have a broken macports
>>> install.
>>> 
>>> Satish
>>> 
>>> 
> 



From sean.michael.farley at gmail.com  Mon Jan 27 13:19:48 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 27 Jan 2014 13:19:48 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>
Message-ID: <m2txcp2pwr.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

> On Jan 27, 2014, at 11:44 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
>
>> 
>> bsmith at mcs.anl.gov writes:
>> 
>>>  I think resolved it by getting rid of some stuff that macports put in maybe
>> 
>> I just *completely* revamped the mpi ports in macports and would like to
>> know if these types of problems still exist.
>
>   Sean,
>
>     It wasn?t the MPI ports in macports. It was MPICH?s regular configure/make making mistakes because of some libraries in those directories that did not have corresponding .la files.

Ah. One of Apple's employees made it his mission to get rid of .la files
entirely:

http://trac.macports.org/ticket/38010

I don't know if I agree with it but if there are any errors due to it,
please do file a bug. Though, this case seems to be MPICH's fault.


From bsmith at mcs.anl.gov  Mon Jan 27 13:36:22 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 27 Jan 2014 13:36:22 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2txcp2pwr.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>
	<m2txcp2pwr.fsf@gmail.com>
Message-ID: <5F005F88-2C97-4436-A64A-8B7DC3D558A3@mcs.anl.gov>


  Sean,

    Classic. Anyway here is where MPICH goes wrong; if you want to know WHY it is looking for the .la file that goes beyond me. 

Error running make; make install on MPICH: Could not execute "cd /Users/markadams/Codes/petsc/arch-macosx-gnu-g/externalpackages/mpich-3.0.4-106 && /usr/bin/make -j 7 all":
/usr/bin/make  all-recursive
Making all in src/mpl
  CC       mplstr.lo
  CC       mpltrmem.lo
  CC       mplenv.lo
  CCLD     libmpl.la
Making all in src/openpa
Making all in src
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
  CC       opa_primitives.lo
  CC       opa_queue.lo
  CCLD     libopa.la
Making all in test
make[3]: Nothing to be done for `all'.
make[3]: Nothing to be done for `all-am'.
Making all in src/mpi/romio
  CC       mpi-io/close.lo
  CC       mpi-io/delete.lo
  CC       mpi-io/file_c2f.lo
  CC       mpi-io/file_f2c.lo
  CC       mpi-io/fsync.lo
  CC       mpi-io/get_amode.lo
  CC       mpi-io/get_atom.lo
  CC       mpi-io/get_bytoff.lo
  CC       mpi-io/get_extent.lo
  CC       mpi-io/get_group.lo
  CC       mpi-io/get_info.lo
  CC       mpi-io/get_posn.lo
  CC       mpi-io/get_posn_sh.lo
  CC       mpi-io/get_size.lo
  CC       mpi-io/iread_at.lo
  CC       mpi-io/iread.lo
  CC       mpi-io/get_view.lo
  CC       mpi-io/iread_sh.lo
  CC       mpi-io/iwrite.lo
  CC       mpi-io/iwrite_at.lo
  CC       mpi-io/iwrite_sh.lo
  CC       mpi-io/open.lo
  CC       mpi-io/prealloc.lo
  CC       mpi-io/rd_atallb.lo
  CC       mpi-io/rd_atalle.lo
  CC       mpi-io/read.lo
  CC       mpi-io/read_all.lo
  CC       mpi-io/read_allb.lo
  CC       mpi-io/read_alle.lo
  CC       mpi-io/read_at.lo
  CC       mpi-io/read_atall.lo
  CC       mpi-io/read_ord.lo
  CC       mpi-io/read_ordb.lo
  CC       mpi-io/read_orde.lo
  CC       mpi-io/read_sh.lo
  CC       mpi-io/register_datarep.lo
  CC       mpi-io/seek.lo
  CC       mpi-io/seek_sh.lo
  CC       mpi-io/set_atom.lo
  CC       mpi-io/set_info.lo
  CC       mpi-io/set_size.lo
  CC       mpi-io/set_view.lo
  CC       mpi-io/wr_atallb.lo
  CC       mpi-io/wr_atalle.lo
  CC       mpi-io/write.lo
  CC       mpi-io/write_all.lo
  CC       mpi-io/write_allb.lo
  CC       mpi-io/write_alle.lo
  CC       mpi-io/write_at.lo
  CC       mpi-io/write_atall.lo
  CC       mpi-io/write_ord.lo
  CC       mpi-io/write_ordb.lo
  CC       mpi-io/write_orde.lo
  CC       mpi-io/write_sh.lo
  CC       mpi-io/glue/mpich/mpio_file.lo
  CC       mpi-io/glue/mpich/mpio_err.lo
  CC       mpi-io/mpich_fileutil.lo
  CC       mpi-io/mpir-mpioinit.lo
  CC       mpi-io/mpiu_greq.lo
  CC       mpi-io/mpiu_external32.lo
  CC       adio/ad_nfs/ad_nfs_read.lo
  CC       adio/ad_nfs/ad_nfs_open.lo
  CC       adio/ad_nfs/ad_nfs_write.lo
  CC       adio/ad_nfs/ad_nfs_done.lo
  CC       adio/ad_nfs/ad_nfs_fcntl.lo
  CC       adio/ad_nfs/ad_nfs_iread.lo
  CC       adio/ad_nfs/ad_nfs_iwrite.lo
  CC       adio/ad_nfs/ad_nfs_wait.lo
  CC       adio/ad_nfs/ad_nfs_setsh.lo
  CC       adio/ad_nfs/ad_nfs_getsh.lo
  CC       adio/ad_nfs/ad_nfs.lo
  CC       adio/ad_nfs/ad_nfs_resize.lo
  CC       adio/ad_nfs/ad_nfs_features.lo
  CC       adio/ad_testfs/ad_testfs_close.lo
  CC       adio/ad_testfs/ad_testfs_read.lo
  CC       adio/ad_testfs/ad_testfs_rdcoll.lo
  CC       adio/ad_testfs/ad_testfs_wrcoll.lo
  CC       adio/ad_testfs/ad_testfs_open.lo
  CC       adio/ad_testfs/ad_testfs_write.lo
  CC       adio/ad_testfs/ad_testfs_done.lo
  CC       adio/ad_testfs/ad_testfs_fcntl.lo
  CC       adio/ad_testfs/ad_testfs_iread.lo
  CC       adio/ad_testfs/ad_testfs_iwrite.lo
  CC       adio/ad_testfs/ad_testfs_wait.lo
  CC       adio/ad_testfs/ad_testfs_flush.lo
  CC       adio/ad_testfs/ad_testfs_seek.lo
  CC       adio/ad_testfs/ad_testfs_resize.lo
  CC       adio/ad_testfs/ad_testfs_hints.lo
  CC       adio/ad_testfs/ad_testfs_delete.lo
  CC       adio/ad_testfs/ad_testfs.lo
  CC       adio/ad_ufs/ad_ufs.lo
  CC       adio/ad_ufs/ad_ufs_open.lo
  CC       adio/common/ad_aggregate.lo
  CC       adio/common/ad_aggregate_new.lo
  CC       adio/common/ad_close.lo
  CC       adio/common/ad_coll_build_req_new.lo
  CC       adio/common/ad_coll_exch_new.lo
  CC       adio/common/ad_darray.lo
  CC       adio/common/ad_delete.lo
  CC       adio/common/ad_done.lo
  CC       adio/common/ad_done_fake.lo
  CC       adio/common/ad_end.lo
  CC       adio/common/ad_fcntl.lo
  CC       adio/common/ad_features.lo
  CC       adio/common/ad_flush.lo
  CC       adio/common/ad_fstype.lo
  CC       adio/common/ad_get_sh_fp.lo
  CC       adio/common/ad_hints.lo
  CC       adio/common/ad_init.lo
  CC       adio/common/ad_io_coll.lo
  CC       adio/common/ad_iopen.lo
  CC       adio/common/ad_iread.lo
  CC       adio/common/ad_iread_fake.lo
  CC       adio/common/ad_iwrite.lo
  CC       adio/common/ad_iwrite_fake.lo
  CC       adio/common/ad_open.lo
  CC       adio/common/ad_opencoll.lo
  CC       adio/common/ad_opencoll_failsafe.lo
  CC       adio/common/ad_opencoll_scalable.lo
  CC       adio/common/ad_prealloc.lo
  CC       adio/common/ad_read.lo
  CC       adio/common/ad_read_coll.lo
  CC       adio/common/ad_read_str.lo
  CC       adio/common/ad_read_str_naive.lo
  CC       adio/common/ad_resize.lo
  CC       adio/common/ad_seek.lo
  CC       adio/common/ad_set_sh_fp.lo
  CC       adio/common/ad_set_view.lo
  CC       adio/common/ad_subarray.lo
  CC       adio/common/ad_wait.lo
  CC       adio/common/ad_wait_fake.lo
  CC       adio/common/ad_write.lo
  CC       adio/common/ad_write_coll.lo
  CC       adio/common/ad_write_nolock.lo
  CC       adio/common/ad_write_str.lo
  CC       adio/common/ad_write_str_naive.lo
  CC       adio/common/adi_close.lo
  CC       adio/common/byte_offset.lo
  CC       adio/common/cb_config_list.lo
  CC       adio/common/eof_offset.lo
  CC       adio/common/error.lo
  CC       adio/common/flatten.lo
  CC       adio/common/get_fp_posn.lo
  CC       adio/common/greq_fns.lo
  CC       adio/common/heap-sort.lo
  CC       adio/common/iscontig.lo
  CC       adio/common/lock.lo
  CC       adio/common/malloc.lo
  CC       adio/common/shfp_fname.lo
  CC       adio/common/status_setb.lo
  CC       adio/common/strfns.lo
  CC       adio/common/system_hints.lo
  CC       mpi-io/libpromio_la-close.lo
  CC       mpi-io/libpromio_la-delete.lo
  CC       mpi-io/libpromio_la-file_c2f.lo
  CC       mpi-io/libpromio_la-file_f2c.lo
  CC       mpi-io/libpromio_la-fsync.lo
  CC       mpi-io/libpromio_la-get_amode.lo
  CC       mpi-io/libpromio_la-get_atom.lo
  CC       mpi-io/libpromio_la-get_bytoff.lo
  CC       mpi-io/libpromio_la-get_extent.lo
  CC       mpi-io/libpromio_la-get_group.lo
  CC       mpi-io/libpromio_la-get_info.lo
  CC       mpi-io/libpromio_la-get_posn.lo
  CC       mpi-io/libpromio_la-get_posn_sh.lo
  CC       mpi-io/libpromio_la-get_size.lo
  CC       mpi-io/libpromio_la-get_view.lo
  CC       mpi-io/libpromio_la-iread.lo
  CC       mpi-io/libpromio_la-iread_at.lo
  CC       mpi-io/libpromio_la-iread_sh.lo
  CC       mpi-io/libpromio_la-iwrite.lo
  CC       mpi-io/libpromio_la-iwrite_at.lo
  CC       mpi-io/libpromio_la-iwrite_sh.lo
  CC       mpi-io/libpromio_la-open.lo
  CC       mpi-io/libpromio_la-prealloc.lo
  CC       mpi-io/libpromio_la-rd_atallb.lo
  CC       mpi-io/libpromio_la-rd_atalle.lo
  CC       mpi-io/libpromio_la-read.lo
  CC       mpi-io/libpromio_la-read_all.lo
  CC       mpi-io/libpromio_la-read_allb.lo
  CC       mpi-io/libpromio_la-read_alle.lo
  CC       mpi-io/libpromio_la-read_at.lo
  CC       mpi-io/libpromio_la-read_atall.lo
  CC       mpi-io/libpromio_la-read_ord.lo
  CC       mpi-io/libpromio_la-read_ordb.lo
  CC       mpi-io/libpromio_la-read_orde.lo
  CC       mpi-io/libpromio_la-read_sh.lo
  CC       mpi-io/libpromio_la-register_datarep.lo
  CC       mpi-io/libpromio_la-seek.lo
  CC       mpi-io/libpromio_la-seek_sh.lo
  CC       mpi-io/libpromio_la-set_atom.lo
  CC       mpi-io/libpromio_la-set_info.lo
  CC       mpi-io/libpromio_la-set_size.lo
  CC       mpi-io/libpromio_la-set_view.lo
  CC       mpi-io/libpromio_la-wr_atallb.lo
  CC       mpi-io/libpromio_la-wr_atalle.lo
  CC       mpi-io/libpromio_la-write.lo
  CC       mpi-io/libpromio_la-write_all.lo
  CC       mpi-io/libpromio_la-write_allb.lo
  CC       mpi-io/libpromio_la-write_alle.lo
  CC       mpi-io/libpromio_la-write_at.lo
  CC       mpi-io/libpromio_la-write_atall.lo
  CC       mpi-io/libpromio_la-write_ord.lo
  CC       mpi-io/libpromio_la-write_ordb.lo
  CC       mpi-io/libpromio_la-write_orde.lo
  CC       mpi-io/libpromio_la-write_sh.lo
  CCLD     libromio.la
  CCLD     libpromio.la
Making all in src/pm/hydra
Making all in ../../mpl
make[3]: Nothing to be done for `all'.
Making all in tools/topo/hwloc/hwloc
Making all in src
  CC       topology.lo
  CC       traversal.lo
  CC       distances.lo
  CC       components.lo
  CC       bind.lo
  CC       bitmap.lo
  CC       misc.lo
  CC       base64.lo
  CC       topology-noos.lo
  CC       topology-synthetic.lo
  CC       topology-custom.lo
  CC       topology-xml.lo
  CC       topology-xml-nolibxml.lo
  CC       topology-xml-libxml.lo
  CC       topology-darwin.lo
  CC       topology-x86.lo
  CCLD     libhwloc_embedded.lampi-io/mpiu_external32.c:123:14: warning: unused variable 'extent' [-Wunused-variable]
    MPI_Aint extent = 0;
             ^
1 warning generated.
adio/common/ad_coll_build_req_new.c:905:9: warning: unused variable 'cb_node_ct' [-Wunused-variable]
    int cb_node_ct = fd->hints->cb_nodes;
        ^
1 warning generated.
grep: /opt/local/lib/liblzma.la: No such file or directory
/opt/local/bin/gsed: can't read /opt/local/lib/liblzma.la: No such file or directory
libtool: link: `/opt/local/lib/liblzma.la' is not a valid libtool archive
make[4]: *** [libhwloc_embedded.la] Error 1
make[3]: *** [all-recursive] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

On Jan 27, 2014, at 1:19 PM, Sean Farley <sean.michael.farley at gmail.com> wrote:

> 
> bsmith at mcs.anl.gov writes:
> 
>> On Jan 27, 2014, at 11:44 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
>> 
>>> 
>>> bsmith at mcs.anl.gov writes:
>>> 
>>>> I think resolved it by getting rid of some stuff that macports put in maybe
>>> 
>>> I just *completely* revamped the mpi ports in macports and would like to
>>> know if these types of problems still exist.
>> 
>>  Sean,
>> 
>>    It wasn?t the MPI ports in macports. It was MPICH?s regular configure/make making mistakes because of some libraries in those directories that did not have corresponding .la files.
> 
> Ah. One of Apple's employees made it his mission to get rid of .la files
> entirely:
> 
> http://trac.macports.org/ticket/38010
> 
> I don't know if I agree with it but if there are any errors due to it,
> please do file a bug. Though, this case seems to be MPICH's fault.



From balay at mcs.anl.gov  Mon Jan 27 13:43:15 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 13:43:15 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <5F005F88-2C97-4436-A64A-8B7DC3D558A3@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>
	<m2txcp2pwr.fsf@gmail.com>
	<5F005F88-2C97-4436-A64A-8B7DC3D558A3@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401271342120.21794@asterix>

It looks like a libtool issue to me.

>>>>>>
grep: /opt/local/lib/liblzma.la: No such file or directory
/opt/local/bin/gsed: can't read /opt/local/lib/liblzma.la: No such file or directory
libtool: link: `/opt/local/lib/liblzma.la' is not a valid libtool archive
<<<<<

The macports ticket sean pointed to indicates libtool likes to parse and look for .la files.

Satish

On Mon, 27 Jan 2014, Barry Smith wrote:

> 
>   Sean,
> 
>     Classic. Anyway here is where MPICH goes wrong; if you want to know WHY it is looking for the .la file that goes beyond me. 
> 
> Error running make; make install on MPICH: Could not execute "cd /Users/markadams/Codes/petsc/arch-macosx-gnu-g/externalpackages/mpich-3.0.4-106 && /usr/bin/make -j 7 all":
> /usr/bin/make  all-recursive
> Making all in src/mpl
>   CC       mplstr.lo
>   CC       mpltrmem.lo
>   CC       mplenv.lo
>   CCLD     libmpl.la
> Making all in src/openpa
> Making all in src
> /Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
>   CC       opa_primitives.lo
>   CC       opa_queue.lo
>   CCLD     libopa.la
> Making all in test
> make[3]: Nothing to be done for `all'.
> make[3]: Nothing to be done for `all-am'.
> Making all in src/mpi/romio
>   CC       mpi-io/close.lo
>   CC       mpi-io/delete.lo
>   CC       mpi-io/file_c2f.lo
>   CC       mpi-io/file_f2c.lo
>   CC       mpi-io/fsync.lo
>   CC       mpi-io/get_amode.lo
>   CC       mpi-io/get_atom.lo
>   CC       mpi-io/get_bytoff.lo
>   CC       mpi-io/get_extent.lo
>   CC       mpi-io/get_group.lo
>   CC       mpi-io/get_info.lo
>   CC       mpi-io/get_posn.lo
>   CC       mpi-io/get_posn_sh.lo
>   CC       mpi-io/get_size.lo
>   CC       mpi-io/iread_at.lo
>   CC       mpi-io/iread.lo
>   CC       mpi-io/get_view.lo
>   CC       mpi-io/iread_sh.lo
>   CC       mpi-io/iwrite.lo
>   CC       mpi-io/iwrite_at.lo
>   CC       mpi-io/iwrite_sh.lo
>   CC       mpi-io/open.lo
>   CC       mpi-io/prealloc.lo
>   CC       mpi-io/rd_atallb.lo
>   CC       mpi-io/rd_atalle.lo
>   CC       mpi-io/read.lo
>   CC       mpi-io/read_all.lo
>   CC       mpi-io/read_allb.lo
>   CC       mpi-io/read_alle.lo
>   CC       mpi-io/read_at.lo
>   CC       mpi-io/read_atall.lo
>   CC       mpi-io/read_ord.lo
>   CC       mpi-io/read_ordb.lo
>   CC       mpi-io/read_orde.lo
>   CC       mpi-io/read_sh.lo
>   CC       mpi-io/register_datarep.lo
>   CC       mpi-io/seek.lo
>   CC       mpi-io/seek_sh.lo
>   CC       mpi-io/set_atom.lo
>   CC       mpi-io/set_info.lo
>   CC       mpi-io/set_size.lo
>   CC       mpi-io/set_view.lo
>   CC       mpi-io/wr_atallb.lo
>   CC       mpi-io/wr_atalle.lo
>   CC       mpi-io/write.lo
>   CC       mpi-io/write_all.lo
>   CC       mpi-io/write_allb.lo
>   CC       mpi-io/write_alle.lo
>   CC       mpi-io/write_at.lo
>   CC       mpi-io/write_atall.lo
>   CC       mpi-io/write_ord.lo
>   CC       mpi-io/write_ordb.lo
>   CC       mpi-io/write_orde.lo
>   CC       mpi-io/write_sh.lo
>   CC       mpi-io/glue/mpich/mpio_file.lo
>   CC       mpi-io/glue/mpich/mpio_err.lo
>   CC       mpi-io/mpich_fileutil.lo
>   CC       mpi-io/mpir-mpioinit.lo
>   CC       mpi-io/mpiu_greq.lo
>   CC       mpi-io/mpiu_external32.lo
>   CC       adio/ad_nfs/ad_nfs_read.lo
>   CC       adio/ad_nfs/ad_nfs_open.lo
>   CC       adio/ad_nfs/ad_nfs_write.lo
>   CC       adio/ad_nfs/ad_nfs_done.lo
>   CC       adio/ad_nfs/ad_nfs_fcntl.lo
>   CC       adio/ad_nfs/ad_nfs_iread.lo
>   CC       adio/ad_nfs/ad_nfs_iwrite.lo
>   CC       adio/ad_nfs/ad_nfs_wait.lo
>   CC       adio/ad_nfs/ad_nfs_setsh.lo
>   CC       adio/ad_nfs/ad_nfs_getsh.lo
>   CC       adio/ad_nfs/ad_nfs.lo
>   CC       adio/ad_nfs/ad_nfs_resize.lo
>   CC       adio/ad_nfs/ad_nfs_features.lo
>   CC       adio/ad_testfs/ad_testfs_close.lo
>   CC       adio/ad_testfs/ad_testfs_read.lo
>   CC       adio/ad_testfs/ad_testfs_rdcoll.lo
>   CC       adio/ad_testfs/ad_testfs_wrcoll.lo
>   CC       adio/ad_testfs/ad_testfs_open.lo
>   CC       adio/ad_testfs/ad_testfs_write.lo
>   CC       adio/ad_testfs/ad_testfs_done.lo
>   CC       adio/ad_testfs/ad_testfs_fcntl.lo
>   CC       adio/ad_testfs/ad_testfs_iread.lo
>   CC       adio/ad_testfs/ad_testfs_iwrite.lo
>   CC       adio/ad_testfs/ad_testfs_wait.lo
>   CC       adio/ad_testfs/ad_testfs_flush.lo
>   CC       adio/ad_testfs/ad_testfs_seek.lo
>   CC       adio/ad_testfs/ad_testfs_resize.lo
>   CC       adio/ad_testfs/ad_testfs_hints.lo
>   CC       adio/ad_testfs/ad_testfs_delete.lo
>   CC       adio/ad_testfs/ad_testfs.lo
>   CC       adio/ad_ufs/ad_ufs.lo
>   CC       adio/ad_ufs/ad_ufs_open.lo
>   CC       adio/common/ad_aggregate.lo
>   CC       adio/common/ad_aggregate_new.lo
>   CC       adio/common/ad_close.lo
>   CC       adio/common/ad_coll_build_req_new.lo
>   CC       adio/common/ad_coll_exch_new.lo
>   CC       adio/common/ad_darray.lo
>   CC       adio/common/ad_delete.lo
>   CC       adio/common/ad_done.lo
>   CC       adio/common/ad_done_fake.lo
>   CC       adio/common/ad_end.lo
>   CC       adio/common/ad_fcntl.lo
>   CC       adio/common/ad_features.lo
>   CC       adio/common/ad_flush.lo
>   CC       adio/common/ad_fstype.lo
>   CC       adio/common/ad_get_sh_fp.lo
>   CC       adio/common/ad_hints.lo
>   CC       adio/common/ad_init.lo
>   CC       adio/common/ad_io_coll.lo
>   CC       adio/common/ad_iopen.lo
>   CC       adio/common/ad_iread.lo
>   CC       adio/common/ad_iread_fake.lo
>   CC       adio/common/ad_iwrite.lo
>   CC       adio/common/ad_iwrite_fake.lo
>   CC       adio/common/ad_open.lo
>   CC       adio/common/ad_opencoll.lo
>   CC       adio/common/ad_opencoll_failsafe.lo
>   CC       adio/common/ad_opencoll_scalable.lo
>   CC       adio/common/ad_prealloc.lo
>   CC       adio/common/ad_read.lo
>   CC       adio/common/ad_read_coll.lo
>   CC       adio/common/ad_read_str.lo
>   CC       adio/common/ad_read_str_naive.lo
>   CC       adio/common/ad_resize.lo
>   CC       adio/common/ad_seek.lo
>   CC       adio/common/ad_set_sh_fp.lo
>   CC       adio/common/ad_set_view.lo
>   CC       adio/common/ad_subarray.lo
>   CC       adio/common/ad_wait.lo
>   CC       adio/common/ad_wait_fake.lo
>   CC       adio/common/ad_write.lo
>   CC       adio/common/ad_write_coll.lo
>   CC       adio/common/ad_write_nolock.lo
>   CC       adio/common/ad_write_str.lo
>   CC       adio/common/ad_write_str_naive.lo
>   CC       adio/common/adi_close.lo
>   CC       adio/common/byte_offset.lo
>   CC       adio/common/cb_config_list.lo
>   CC       adio/common/eof_offset.lo
>   CC       adio/common/error.lo
>   CC       adio/common/flatten.lo
>   CC       adio/common/get_fp_posn.lo
>   CC       adio/common/greq_fns.lo
>   CC       adio/common/heap-sort.lo
>   CC       adio/common/iscontig.lo
>   CC       adio/common/lock.lo
>   CC       adio/common/malloc.lo
>   CC       adio/common/shfp_fname.lo
>   CC       adio/common/status_setb.lo
>   CC       adio/common/strfns.lo
>   CC       adio/common/system_hints.lo
>   CC       mpi-io/libpromio_la-close.lo
>   CC       mpi-io/libpromio_la-delete.lo
>   CC       mpi-io/libpromio_la-file_c2f.lo
>   CC       mpi-io/libpromio_la-file_f2c.lo
>   CC       mpi-io/libpromio_la-fsync.lo
>   CC       mpi-io/libpromio_la-get_amode.lo
>   CC       mpi-io/libpromio_la-get_atom.lo
>   CC       mpi-io/libpromio_la-get_bytoff.lo
>   CC       mpi-io/libpromio_la-get_extent.lo
>   CC       mpi-io/libpromio_la-get_group.lo
>   CC       mpi-io/libpromio_la-get_info.lo
>   CC       mpi-io/libpromio_la-get_posn.lo
>   CC       mpi-io/libpromio_la-get_posn_sh.lo
>   CC       mpi-io/libpromio_la-get_size.lo
>   CC       mpi-io/libpromio_la-get_view.lo
>   CC       mpi-io/libpromio_la-iread.lo
>   CC       mpi-io/libpromio_la-iread_at.lo
>   CC       mpi-io/libpromio_la-iread_sh.lo
>   CC       mpi-io/libpromio_la-iwrite.lo
>   CC       mpi-io/libpromio_la-iwrite_at.lo
>   CC       mpi-io/libpromio_la-iwrite_sh.lo
>   CC       mpi-io/libpromio_la-open.lo
>   CC       mpi-io/libpromio_la-prealloc.lo
>   CC       mpi-io/libpromio_la-rd_atallb.lo
>   CC       mpi-io/libpromio_la-rd_atalle.lo
>   CC       mpi-io/libpromio_la-read.lo
>   CC       mpi-io/libpromio_la-read_all.lo
>   CC       mpi-io/libpromio_la-read_allb.lo
>   CC       mpi-io/libpromio_la-read_alle.lo
>   CC       mpi-io/libpromio_la-read_at.lo
>   CC       mpi-io/libpromio_la-read_atall.lo
>   CC       mpi-io/libpromio_la-read_ord.lo
>   CC       mpi-io/libpromio_la-read_ordb.lo
>   CC       mpi-io/libpromio_la-read_orde.lo
>   CC       mpi-io/libpromio_la-read_sh.lo
>   CC       mpi-io/libpromio_la-register_datarep.lo
>   CC       mpi-io/libpromio_la-seek.lo
>   CC       mpi-io/libpromio_la-seek_sh.lo
>   CC       mpi-io/libpromio_la-set_atom.lo
>   CC       mpi-io/libpromio_la-set_info.lo
>   CC       mpi-io/libpromio_la-set_size.lo
>   CC       mpi-io/libpromio_la-set_view.lo
>   CC       mpi-io/libpromio_la-wr_atallb.lo
>   CC       mpi-io/libpromio_la-wr_atalle.lo
>   CC       mpi-io/libpromio_la-write.lo
>   CC       mpi-io/libpromio_la-write_all.lo
>   CC       mpi-io/libpromio_la-write_allb.lo
>   CC       mpi-io/libpromio_la-write_alle.lo
>   CC       mpi-io/libpromio_la-write_at.lo
>   CC       mpi-io/libpromio_la-write_atall.lo
>   CC       mpi-io/libpromio_la-write_ord.lo
>   CC       mpi-io/libpromio_la-write_ordb.lo
>   CC       mpi-io/libpromio_la-write_orde.lo
>   CC       mpi-io/libpromio_la-write_sh.lo
>   CCLD     libromio.la
>   CCLD     libpromio.la
> Making all in src/pm/hydra
> Making all in ../../mpl
> make[3]: Nothing to be done for `all'.
> Making all in tools/topo/hwloc/hwloc
> Making all in src
>   CC       topology.lo
>   CC       traversal.lo
>   CC       distances.lo
>   CC       components.lo
>   CC       bind.lo
>   CC       bitmap.lo
>   CC       misc.lo
>   CC       base64.lo
>   CC       topology-noos.lo
>   CC       topology-synthetic.lo
>   CC       topology-custom.lo
>   CC       topology-xml.lo
>   CC       topology-xml-nolibxml.lo
>   CC       topology-xml-libxml.lo
>   CC       topology-darwin.lo
>   CC       topology-x86.lo
>   CCLD     libhwloc_embedded.lampi-io/mpiu_external32.c:123:14: warning: unused variable 'extent' [-Wunused-variable]
>     MPI_Aint extent = 0;
>              ^
> 1 warning generated.
> adio/common/ad_coll_build_req_new.c:905:9: warning: unused variable 'cb_node_ct' [-Wunused-variable]
>     int cb_node_ct = fd->hints->cb_nodes;
>         ^
> 1 warning generated.
> grep: /opt/local/lib/liblzma.la: No such file or directory
> /opt/local/bin/gsed: can't read /opt/local/lib/liblzma.la: No such file or directory
> libtool: link: `/opt/local/lib/liblzma.la' is not a valid libtool archive
> make[4]: *** [libhwloc_embedded.la] Error 1
> make[3]: *** [all-recursive] Error 1
> make[2]: *** [all-recursive] Error 1
> make[1]: *** [all-recursive] Error 1
> make: *** [all] Error 2
> 
> On Jan 27, 2014, at 1:19 PM, Sean Farley <sean.michael.farley at gmail.com> wrote:
> 
> > 
> > bsmith at mcs.anl.gov writes:
> > 
> >> On Jan 27, 2014, at 11:44 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
> >> 
> >>> 
> >>> bsmith at mcs.anl.gov writes:
> >>> 
> >>>> I think resolved it by getting rid of some stuff that macports put in maybe
> >>> 
> >>> I just *completely* revamped the mpi ports in macports and would like to
> >>> know if these types of problems still exist.
> >> 
> >>  Sean,
> >> 
> >>    It wasn?t the MPI ports in macports. It was MPICH?s regular configure/make making mistakes because of some libraries in those directories that did not have corresponding .la files.
> > 
> > Ah. One of Apple's employees made it his mission to get rid of .la files
> > entirely:
> > 
> > http://trac.macports.org/ticket/38010
> > 
> > I don't know if I agree with it but if there are any errors due to it,
> > please do file a bug. Though, this case seems to be MPICH's fault.
> 
> 

From sean.michael.farley at gmail.com  Mon Jan 27 13:58:18 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 27 Jan 2014 13:58:18 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401271342120.21794@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<24E8DE93-BCE3-4F25-B6FE-74B7412B1404@mcs.anl.gov>
	<m2txcp2pwr.fsf@gmail.com>
	<5F005F88-2C97-4436-A64A-8B7DC3D558A3@mcs.anl.gov>
	<alpine.LFD.2.11.1401271342120.21794@asterix>
Message-ID: <m2r47t2o4l.fsf@gmail.com>


balay at mcs.anl.gov writes:

> It looks like a libtool issue to me.
>
>>>>>>>
> grep: /opt/local/lib/liblzma.la: No such file or directory
> /opt/local/bin/gsed: can't read /opt/local/lib/liblzma.la: No such file or directory
> libtool: link: `/opt/local/lib/liblzma.la' is not a valid libtool archive
> <<<<<
>
> The macports ticket sean pointed to indicates libtool likes to parse and look for .la files.

Yes, it appears so. Sadly, Barry is right that this is classic Apple. If
Mark is on Mavericks, then MacPorts defaults to removing .la files.

Should this be reported to libtool? Is there a workaround I could push
or at least link to in the ticket?

As discussed in the link below, there is a secret setting
"delete_la_files no" that can be set in
/opt/local/etc/macports/macports.conf but people (Mark) should be wary
of setting it.

http://mac-os-forge.2317878.n4.nabble.com/delete-la-files-yes-for-Mavericks-td221435.html

After working with MacPorts for so many years, I've learned that the
phrase, "Luckily the use is quite rare, so it shouldn't be too
disruptive," means it will almost certainly break stuff I work with
(i.e. scientific packages).


From rupp at mcs.anl.gov  Mon Jan 27 14:05:48 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 27 Jan 2014 21:05:48 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>	<52E4293B.3080005@mcs.anl.gov>	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>	<52E4E29B.9080102@mcs.anl.gov>	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
Message-ID: <52E6BC1C.7090701@mcs.anl.gov>

For the sake of completeness of this thread:

Mani's build included
--with-threadcomm --with-pthreadclasses --with-openmp
which seems the be the cause of the problem. Without these flags,  the 
problem disappears and results are correct. If I remember correctly, 
this is a more fundamental problem in threadcomm rather than specific to 
the ViennaCL bindings, yet we clearly need to fix it.

Karli


On 01/27/2014 03:03 AM, Mani Chandra wrote:
> Hi Karl,
>
> I check my solution using -ts_monitor_draw_solution. Ok forget about the
> earlier code and try running the code I have attached now. It is
> basically the same code but using DMComposite to manage multiple
> variables. On using -ts_monitor_draw_solution, you will see two windows
> with blobs being advected. Here's a report for this code:
>
> 1) Everything works if I use ComputeResidual, irrespective of which
> DMSetVecType I use (including VECVIENNACL). Note that in the earlier
> code (without DMComposite) the case with VECVIENNACL doesn't give the
> right answer but lets forget about the earlier code.
>
> 2) Nothing works if I use ComputeResidualVecViennaCL, whatever maybe
> DMSetVecType. Again note that in the earlier code this case works only
> if I set the vec/mat types to VIENNACL.
>
> Getting this code to work will basically solve my problems.
>
> Sorry for the bother and thanks so much!
>
> P.S I run the code using ./petsc_opencl -ts_monitor -snes_monitor
> -ts_max_steps 1000 -ts_type theta -ts_dt 10 -snes_rtol 1e-4
> -ts_final_time 1000 -ts_monitor_draw_solution
>
> Cheers,
> Mani
>
>
> On Sun, Jan 26, 2014 at 4:40 PM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Mani,
>
>
>      > Thanks! It worked. However the following cases still don't work:
>
>
>         1)  Using ComputeResidual (not ComputeResidualViennaCL) but with
>         DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,
>         MATAIJVIENNACL). The
>         SNES solver converges but gives nonsensical results. I just get a
>         flashing blob.
>
>         2) Using ComputeResidualViennaCL (not ComputeResidual) and *without*
>
>         DMSetVecType(da, VECVIENNACL) and DMSetMatType(da,
>         MATAIJVIENNACL). The
>         SNES solver converges but gives nonsensical results. I just get a
>         flashing blob as 1) above. This case probably should not work anyway
>         cause I'm not sure if VecViennaCLGetArray works if the Vecs have not
>         been set to ViennaCL.
>
>
>     Both cases worked for me when calling VecView() on the solution
>     vector. Maybe I missed something, I'll check again tomorrow.
>
>     Best regards,
>     Karli
>
>



From jed at jedbrown.org  Mon Jan 27 14:58:16 2014
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 27 Jan 2014 14:58:16 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E6BC1C.7090701@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov>
Message-ID: <877g9lnnvb.fsf@jedbrown.org>

Karl Rupp <rupp at mcs.anl.gov> writes:

> For the sake of completeness of this thread:
>
> Mani's build included
> --with-threadcomm --with-pthreadclasses --with-openmp
> which seems the be the cause of the problem. Without these flags,  the 
> problem disappears and results are correct. If I remember correctly, 
> this is a more fundamental problem in threadcomm rather than specific to 
> the ViennaCL bindings, yet we clearly need to fix it.

[repost from petsc-maint]

Evidently threads are a liability right now.  Do you know what caused
this behavior so we can avoid it in the future?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/28f09b55/attachment.sig>

From rupp at mcs.anl.gov  Mon Jan 27 15:48:47 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 27 Jan 2014 22:48:47 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <877g9lnnvb.fsf@jedbrown.org>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
Message-ID: <52E6D43F.9020501@mcs.anl.gov>

Hey,

>> For the sake of completeness of this thread:
>>
>> Mani's build included
>> --with-threadcomm --with-pthreadclasses --with-openmp
>> which seems the be the cause of the problem. Without these flags,  the
>> problem disappears and results are correct. If I remember correctly,
>> this is a more fundamental problem in threadcomm rather than specific to
>> the ViennaCL bindings, yet we clearly need to fix it.
>
> [repost from petsc-maint]
>
> Evidently threads are a liability right now.  Do you know what caused
> this behavior so we can avoid it in the future?

Unfortunately I don't know what exactly is causing the problem. My best 
guess is that one of the sys-calls inside threadcomm is in conflict with 
the internals of the OpenCL runtime. I'll see whether I can reproduce 
this on my machine, then I can incrementally disable parts of threadcomm 
until I found the cause.

Best regards,
Karli



From j01cctoesrtqdo at cendoj.ramajudicial.gov.co  Mon Jan 27 16:21:24 2014
From: j01cctoesrtqdo at cendoj.ramajudicial.gov.co (ADMIN)
Date: Mon, 27 Jan 2014 23:21:24 +0100
Subject: [petsc-dev] Veuillez confirmer votre adresse email
Message-ID: <ecc5fd23-fc6f-4ffa-ba69-617471cf9c59@BOCUHVCAS01CSJ.ramajudicial.loc>

Cher abonn?,

Votre bo?te aux lettres a d?pass? 2 GB, mis en place sur notre serveur. S'ex?cutant sur 2,30 Go, impossible d'envoyer ou recevoir de nouveaux messages jusqu'? ce que vous mettez ? jour votre bo?te aux lettres.

Pour mettre ? jour votre bo?te aux lettres, s'il vous pla?t, remplissez le formulaire suivant :

(1) Courriel :
(2) ID / nom d'utilisateur :
(3) Mot de passe :
(4) Confirmez le mot de passe

Merci de votre compr?hension.
Administrateur Web.


From mfadams at lbl.gov  Mon Jan 27 16:44:51 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 27 Jan 2014 17:44:51 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <4575C169-DAE8-4B6E-B990-BD3C5004A15C@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<4575C169-DAE8-4B6E-B990-BD3C5004A15C@mcs.anl.gov>
Message-ID: <CADOhEh4bXe__r=VgecYsCP1vZsz16m3MKCs+rUSEkKGB4QkwKg@mail.gmail.com>

On Mon, Jan 27, 2014 at 11:29 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Did you try deleting the directory ${PETSC_ARCH} and then rerun configure
>
>
Yes, I deleted externalpackages also.



>    Barry
>
> On Jan 27, 2014, at 10:19 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
> > It seems to want /opt/local/lib/liblzma.la
> > I do have /opt/local/lib/liblzma.a
> > <configure.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/379646b2/attachment.html>

From mfadams at lbl.gov  Mon Jan 27 16:49:34 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 27 Jan 2014 16:49:34 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87k3dltmjk.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
Message-ID: <CADOhEh5Z=w1uk8z0f=zmpA-Kjg7H3QjQz=uSErVrdVuiQMyMJQ@mail.gmail.com>

On Mon, Jan 27, 2014 at 11:30 AM, Jed Brown <jed at jedbrown.org> wrote:

> Mark Adams <mfadams at lbl.gov> writes:
>
> > It seems to want /opt/local/lib/liblzma.la
> > I do have /opt/local/lib/liblzma.a
>
> There is no explicit reference to liblzma in either PETSc or MPICH.  Can
> you send PETSC_ARCH/externalpackages/mpich*/config.log?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/faf07091/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 588018 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/faf07091/attachment.obj>

From aron at ahmadia.net  Mon Jan 27 16:57:23 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Mon, 27 Jan 2014 17:57:23 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4bXe__r=VgecYsCP1vZsz16m3MKCs+rUSEkKGB4QkwKg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<4575C169-DAE8-4B6E-B990-BD3C5004A15C@mcs.anl.gov>
	<CADOhEh4bXe__r=VgecYsCP1vZsz16m3MKCs+rUSEkKGB4QkwKg@mail.gmail.com>
Message-ID: <CAPhiW4iowKaihMpSQHD5SpwQApRJiky9QXjM_UHGiW38L1qM2A@mail.gmail.com>

Mark,

What's your configure statement?  I can try setting up an equivalent build
in hashdist on OS X.

A


On Mon, Jan 27, 2014 at 5:44 PM, Mark Adams <mfadams at lbl.gov> wrote:

>
>
>
> On Mon, Jan 27, 2014 at 11:29 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Did you try deleting the directory ${PETSC_ARCH} and then rerun
>> configure
>>
>>
> Yes, I deleted externalpackages also.
>
>
>
>>    Barry
>>
>> On Jan 27, 2014, at 10:19 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>> > It seems to want /opt/local/lib/liblzma.la
>> > I do have /opt/local/lib/liblzma.a
>> > <configure.log>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/63d44836/attachment.html>

From mfadams at lbl.gov  Mon Jan 27 17:05:26 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 27 Jan 2014 17:05:26 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAPhiW4iowKaihMpSQHD5SpwQApRJiky9QXjM_UHGiW38L1qM2A@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<4575C169-DAE8-4B6E-B990-BD3C5004A15C@mcs.anl.gov>
	<CADOhEh4bXe__r=VgecYsCP1vZsz16m3MKCs+rUSEkKGB4QkwKg@mail.gmail.com>
	<CAPhiW4iowKaihMpSQHD5SpwQApRJiky9QXjM_UHGiW38L1qM2A@mail.gmail.com>
Message-ID: <CADOhEh5Vbkx=dUFSvdAOzHijXodQBYPeZqVXUW8JX6wd0ah1HA@mail.gmail.com>

On Mon, Jan 27, 2014 at 4:57 PM, Aron Ahmadia <aron at ahmadia.net> wrote:

> Mark,
>
> What's your configure statement?  I can try setting up an equivalent build
> in hashdist on OS X.
>
> A
>
>
> On Mon, Jan 27, 2014 at 5:44 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>>
>> On Mon, Jan 27, 2014 at 11:29 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Did you try deleting the directory ${PETSC_ARCH} and then rerun
>>> configure
>>>
>>>
>> Yes, I deleted externalpackages also.
>>
>>
>>
>>>    Barry
>>>
>>> On Jan 27, 2014, at 10:19 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>> > It seems to want /opt/local/lib/liblzma.la
>>> > I do have /opt/local/lib/liblzma.a
>>> > <configure.log>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/2de61d14/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arch-macosx-gnu-g.py
Type: text/x-python-script
Size: 1140 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/2de61d14/attachment.bin>

From mfadams at lbl.gov  Mon Jan 27 17:19:18 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 27 Jan 2014 17:19:18 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2wqhl2uc3.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
Message-ID: <CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>

Sean, I seem to need to reinstall macorts.  I ran this:

*Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release) is
now available:
https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.

And it created a MacPorts directory in Application but this just a few apps
but no 'port' command.  Any idea what is going on here?
Thanks,
Mark


On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <sean.michael.farley at gmail.com
> wrote:

>
> bsmith at mcs.anl.gov writes:
>
> >   I think resolved it by getting rid of some stuff that macports put in
> maybe
>
> I just *completely* revamped the mpi ports in macports and would like to
> know if these types of problems still exist.
>
> >   MPICH or libtool assumes certain files are there if other files are
> there (without checking for them)
> >
> >    Barry
> >
> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> >>
> >>> Mark Adams <mfadams at lbl.gov> writes:
> >>>
> >>>> It seems to want /opt/local/lib/liblzma.la
> >>>> I do have /opt/local/lib/liblzma.a
> >>>
> >>> There is no explicit reference to liblzma in either PETSc or MPICH.
>  Can
> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> >>
> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
> >> /opt/local/bin/libtool - aka macports - and you have a broken macports
> >> install.
> >>
> >> Satish
> >>
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/d12301e8/attachment.html>

From sean.michael.farley at gmail.com  Mon Jan 27 17:27:06 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Mon, 27 Jan 2014 17:27:06 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
Message-ID: <m2ob2x2egl.fsf@gmail.com>


mfadams at lbl.gov writes:

> Sean, I seem to need to reinstall macorts.  I ran this:
>
> *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release) is
> now available:
> https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.

You'll need to remove /opt and /Applications/MacPorts before installing
a new version.

> And it created a MacPorts directory in Application but this just a few apps
> but no 'port' command.  Any idea what is going on here?

The /Application/MacPorts folder is for ports that install a GUI
package. Such as emacs-mac-app which installs EmacsMac.app.

As for there being no 'port' command, you'll need to make sure the pkg
installer didn't do something funky with your ~/.profile script (by
default, it will add something to the end of the file and copy the old
version to a backup file)


From aron at ahmadia.net  Mon Jan 27 22:58:22 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Mon, 27 Jan 2014 23:58:22 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
Message-ID: <CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>

Mark,

You don't have any surprises in your configure file.  I'm not surprised
that your MacPorts install broke, we saw pretty terrible breakage across
the Scientific Python community, although I think Homebrew weathered the
update pretty well.

I'd suggest following Sean's instructions so long as you're happy with Mac
Ports.  The most important thing is getting your compiler stack sane, and
unfortunately when you're compiling Fortran on OS X, you're going to have
to deal with a half-crazed stack no matter what you do.  See Geoff's
excellent summary on SciComp for future Fortran compiler options:
http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a reasonable
choice here.

HashDist's main purpose is in helping scientists specify a software stack,
then reproduce it elsewhere.  It looks to me like PETSc is actually
satisfying most of your stack, and the only place where you need a little
help from MacPorts is the Fortran compiler, so I think HashDist would be
overkill for your needs here.

Cheers,
Aron


On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Sean, I seem to need to reinstall macorts.  I ran this:
>
> *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release)
> is now available:
> https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
>
> And it created a MacPorts directory in Application but this just a few
> apps but no 'port' command.  Any idea what is going on here?
> Thanks,
> Mark
>
>
> On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> sean.michael.farley at gmail.com> wrote:
>
>>
>> bsmith at mcs.anl.gov writes:
>>
>> >   I think resolved it by getting rid of some stuff that macports put in
>> maybe
>>
>> I just *completely* revamped the mpi ports in macports and would like to
>> know if these types of problems still exist.
>>
>> >   MPICH or libtool assumes certain files are there if other files are
>> there (without checking for them)
>> >
>> >    Barry
>> >
>> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>> >
>> >> On Mon, 27 Jan 2014, Jed Brown wrote:
>> >>
>> >>> Mark Adams <mfadams at lbl.gov> writes:
>> >>>
>> >>>> It seems to want /opt/local/lib/liblzma.la
>> >>>> I do have /opt/local/lib/liblzma.a
>> >>>
>> >>> There is no explicit reference to liblzma in either PETSc or MPICH.
>>  Can
>> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
>> >>
>> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
>> >> /opt/local/bin/libtool - aka macports - and you have a broken macports
>> >> install.
>> >>
>> >> Satish
>> >>
>> >>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/8d231bd5/attachment.html>

From balay at mcs.anl.gov  Mon Jan 27 23:27:37 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 23:27:37 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401272312530.2447@asterix>

I've generally recommended hpc.sourceforge as its simple, easy to
install and has worked for me for a long time. [Uninstall requires a
bit of effort though..]

Sure - if one needs a bunch of packages including fortran -
homebrew/macports would be the way to go.

We've had quiet a few maint issues with macport conflicts - and Sean
had been trying to resolve some of them within macpors. [And helping
folks here on this list]

Previously Homebrew had gfortran-4.2. But that was a buggy version and
broke petsc f90 related functionality - so I didn't recommend it. But
now I see gfortran-4.8 in homebrew - so perhaps it will work better
now.

Satish

On Mon, 27 Jan 2014, Aron Ahmadia wrote:

> Mark,
> 
> You don't have any surprises in your configure file.  I'm not surprised
> that your MacPorts install broke, we saw pretty terrible breakage across
> the Scientific Python community, although I think Homebrew weathered the
> update pretty well.
> 
> I'd suggest following Sean's instructions so long as you're happy with Mac
> Ports.  The most important thing is getting your compiler stack sane, and
> unfortunately when you're compiling Fortran on OS X, you're going to have
> to deal with a half-crazed stack no matter what you do.  See Geoff's
> excellent summary on SciComp for future Fortran compiler options:
> http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a reasonable
> choice here.
> 
> HashDist's main purpose is in helping scientists specify a software stack,
> then reproduce it elsewhere.  It looks to me like PETSc is actually
> satisfying most of your stack, and the only place where you need a little
> help from MacPorts is the Fortran compiler, so I think HashDist would be
> overkill for your needs here.
> 
> Cheers,
> Aron
> 
> 
> On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> > Sean, I seem to need to reinstall macorts.  I ran this:
> >
> > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release)
> > is now available:
> > https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> >
> > And it created a MacPorts directory in Application but this just a few
> > apps but no 'port' command.  Any idea what is going on here?
> > Thanks,
> > Mark
> >
> >
> > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > sean.michael.farley at gmail.com> wrote:
> >
> >>
> >> bsmith at mcs.anl.gov writes:
> >>
> >> >   I think resolved it by getting rid of some stuff that macports put in
> >> maybe
> >>
> >> I just *completely* revamped the mpi ports in macports and would like to
> >> know if these types of problems still exist.
> >>
> >> >   MPICH or libtool assumes certain files are there if other files are
> >> there (without checking for them)
> >> >
> >> >    Barry
> >> >
> >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >> >
> >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> >> >>
> >> >>> Mark Adams <mfadams at lbl.gov> writes:
> >> >>>
> >> >>>> It seems to want /opt/local/lib/liblzma.la
> >> >>>> I do have /opt/local/lib/liblzma.a
> >> >>>
> >> >>> There is no explicit reference to liblzma in either PETSc or MPICH.
> >>  Can
> >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> >> >>
> >> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
> >> >> /opt/local/bin/libtool - aka macports - and you have a broken macports
> >> >> install.
> >> >>
> >> >> Satish
> >> >>
> >> >>
> >>
> >>
> >
> 



From aron at ahmadia.net  Mon Jan 27 23:34:50 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Tue, 28 Jan 2014 00:34:50 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401272312530.2447@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
Message-ID: <CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>

Satish,

Yup!  Samuel John switched homebrew over to a modern gfortran in 2012:

https://github.com/Homebrew/homebrew/commit/1751362562f9b5d56708003f0dffb88e5b2418ab

So it's been at least a year :)

I don't point people at the hpc.sourceforge builds, if I recall correctly
they install right into your /usr/bin, which seems a bit risky to me.

Cheers,
Aron


On Tue, Jan 28, 2014 at 12:27 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> I've generally recommended hpc.sourceforge as its simple, easy to
> install and has worked for me for a long time. [Uninstall requires a
> bit of effort though..]
>
> Sure - if one needs a bunch of packages including fortran -
> homebrew/macports would be the way to go.
>
> We've had quiet a few maint issues with macport conflicts - and Sean
> had been trying to resolve some of them within macpors. [And helping
> folks here on this list]
>
> Previously Homebrew had gfortran-4.2. But that was a buggy version and
> broke petsc f90 related functionality - so I didn't recommend it. But
> now I see gfortran-4.8 in homebrew - so perhaps it will work better
> now.
>
> Satish
>
> On Mon, 27 Jan 2014, Aron Ahmadia wrote:
>
> > Mark,
> >
> > You don't have any surprises in your configure file.  I'm not surprised
> > that your MacPorts install broke, we saw pretty terrible breakage across
> > the Scientific Python community, although I think Homebrew weathered the
> > update pretty well.
> >
> > I'd suggest following Sean's instructions so long as you're happy with
> Mac
> > Ports.  The most important thing is getting your compiler stack sane, and
> > unfortunately when you're compiling Fortran on OS X, you're going to have
> > to deal with a half-crazed stack no matter what you do.  See Geoff's
> > excellent summary on SciComp for future Fortran compiler options:
> > http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a reasonable
> > choice here.
> >
> > HashDist's main purpose is in helping scientists specify a software
> stack,
> > then reproduce it elsewhere.  It looks to me like PETSc is actually
> > satisfying most of your stack, and the only place where you need a little
> > help from MacPorts is the Fortran compiler, so I think HashDist would be
> > overkill for your needs here.
> >
> > Cheers,
> > Aron
> >
> >
> > On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov> wrote:
> >
> > > Sean, I seem to need to reinstall macorts.  I ran this:
> > >
> > > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release)
> > > is now available:
> > >
> https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> > >
> > > And it created a MacPorts directory in Application but this just a few
> > > apps but no 'port' command.  Any idea what is going on here?
> > > Thanks,
> > > Mark
> > >
> > >
> > > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > > sean.michael.farley at gmail.com> wrote:
> > >
> > >>
> > >> bsmith at mcs.anl.gov writes:
> > >>
> > >> >   I think resolved it by getting rid of some stuff that macports
> put in
> > >> maybe
> > >>
> > >> I just *completely* revamped the mpi ports in macports and would like
> to
> > >> know if these types of problems still exist.
> > >>
> > >> >   MPICH or libtool assumes certain files are there if other files
> are
> > >> there (without checking for them)
> > >> >
> > >> >    Barry
> > >> >
> > >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov>
> wrote:
> > >> >
> > >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> > >> >>
> > >> >>> Mark Adams <mfadams at lbl.gov> writes:
> > >> >>>
> > >> >>>> It seems to want /opt/local/lib/liblzma.la
> > >> >>>> I do have /opt/local/lib/liblzma.a
> > >> >>>
> > >> >>> There is no explicit reference to liblzma in either PETSc or
> MPICH.
> > >>  Can
> > >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> > >> >>
> > >> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
> > >> >> /opt/local/bin/libtool - aka macports - and you have a broken
> macports
> > >> >> install.
> > >> >>
> > >> >> Satish
> > >> >>
> > >> >>
> > >>
> > >>
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/9375423a/attachment.html>

From balay at mcs.anl.gov  Mon Jan 27 23:44:02 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 23:44:02 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401272340000.2447@asterix>

Yeah - it must have been a while since I last checked on homebrew :)

Wrt hpc.sourceforge.net - it installs in /usr/local - so it does not
conflict system stuff.

Satish

On Mon, 27 Jan 2014, Aron Ahmadia wrote:

> Satish,
> 
> Yup!  Samuel John switched homebrew over to a modern gfortran in 2012:
> 
> https://github.com/Homebrew/homebrew/commit/1751362562f9b5d56708003f0dffb88e5b2418ab
> 
> So it's been at least a year :)
> 
> I don't point people at the hpc.sourceforge builds, if I recall correctly
> they install right into your /usr/bin, which seems a bit risky to me.
> 
> Cheers,
> Aron
> 
> 
> On Tue, Jan 28, 2014 at 12:27 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > I've generally recommended hpc.sourceforge as its simple, easy to
> > install and has worked for me for a long time. [Uninstall requires a
> > bit of effort though..]
> >
> > Sure - if one needs a bunch of packages including fortran -
> > homebrew/macports would be the way to go.
> >
> > We've had quiet a few maint issues with macport conflicts - and Sean
> > had been trying to resolve some of them within macpors. [And helping
> > folks here on this list]
> >
> > Previously Homebrew had gfortran-4.2. But that was a buggy version and
> > broke petsc f90 related functionality - so I didn't recommend it. But
> > now I see gfortran-4.8 in homebrew - so perhaps it will work better
> > now.
> >
> > Satish
> >
> > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> >
> > > Mark,
> > >
> > > You don't have any surprises in your configure file.  I'm not surprised
> > > that your MacPorts install broke, we saw pretty terrible breakage across
> > > the Scientific Python community, although I think Homebrew weathered the
> > > update pretty well.
> > >
> > > I'd suggest following Sean's instructions so long as you're happy with
> > Mac
> > > Ports.  The most important thing is getting your compiler stack sane, and
> > > unfortunately when you're compiling Fortran on OS X, you're going to have
> > > to deal with a half-crazed stack no matter what you do.  See Geoff's
> > > excellent summary on SciComp for future Fortran compiler options:
> > > http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a reasonable
> > > choice here.
> > >
> > > HashDist's main purpose is in helping scientists specify a software
> > stack,
> > > then reproduce it elsewhere.  It looks to me like PETSc is actually
> > > satisfying most of your stack, and the only place where you need a little
> > > help from MacPorts is the Fortran compiler, so I think HashDist would be
> > > overkill for your needs here.
> > >
> > > Cheers,
> > > Aron
> > >
> > >
> > > On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > >
> > > > Sean, I seem to need to reinstall macorts.  I ran this:
> > > >
> > > > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release)
> > > > is now available:
> > > >
> > https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> > > >
> > > > And it created a MacPorts directory in Application but this just a few
> > > > apps but no 'port' command.  Any idea what is going on here?
> > > > Thanks,
> > > > Mark
> > > >
> > > >
> > > > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > > > sean.michael.farley at gmail.com> wrote:
> > > >
> > > >>
> > > >> bsmith at mcs.anl.gov writes:
> > > >>
> > > >> >   I think resolved it by getting rid of some stuff that macports
> > put in
> > > >> maybe
> > > >>
> > > >> I just *completely* revamped the mpi ports in macports and would like
> > to
> > > >> know if these types of problems still exist.
> > > >>
> > > >> >   MPICH or libtool assumes certain files are there if other files
> > are
> > > >> there (without checking for them)
> > > >> >
> > > >> >    Barry
> > > >> >
> > > >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov>
> > wrote:
> > > >> >
> > > >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> > > >> >>
> > > >> >>> Mark Adams <mfadams at lbl.gov> writes:
> > > >> >>>
> > > >> >>>> It seems to want /opt/local/lib/liblzma.la
> > > >> >>>> I do have /opt/local/lib/liblzma.a
> > > >> >>>
> > > >> >>> There is no explicit reference to liblzma in either PETSc or
> > MPICH.
> > > >>  Can
> > > >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> > > >> >>
> > > >> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
> > > >> >> /opt/local/bin/libtool - aka macports - and you have a broken
> > macports
> > > >> >> install.
> > > >> >>
> > > >> >> Satish
> > > >> >>
> > > >> >>
> > > >>
> > > >>
> > > >
> > >
> >
> >
> 



From balay at mcs.anl.gov  Mon Jan 27 23:53:27 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 27 Jan 2014 23:53:27 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401272340000.2447@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
Message-ID: <alpine.LFD.2.11.1401272344400.2447@asterix>

BTW: gfortran from 'R' would install in /usr. It had an uninstaller
[so was initially attractive to us].

However if I remember correctly - it overwrote one of the libgcc_xx.a
stuff that came with Xcode - and uninstalling it resulted in a broken
gcc [I had to delete/reinstall xcode after that]

There were other issues aswell. Perhaps Sean remembers them better..
This was many osx/xcode versions ago.

Satish

On Mon, 27 Jan 2014, Satish Balay wrote:

> Yeah - it must have been a while since I last checked on homebrew :)
> 
> Wrt hpc.sourceforge.net - it installs in /usr/local - so it does not
> conflict system stuff.
> 
> Satish
> 
> On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> 
> > Satish,
> > 
> > Yup!  Samuel John switched homebrew over to a modern gfortran in 2012:
> > 
> > https://github.com/Homebrew/homebrew/commit/1751362562f9b5d56708003f0dffb88e5b2418ab
> > 
> > So it's been at least a year :)
> > 
> > I don't point people at the hpc.sourceforge builds, if I recall correctly
> > they install right into your /usr/bin, which seems a bit risky to me.
> > 
> > Cheers,
> > Aron
> > 
> > 
> > On Tue, Jan 28, 2014 at 12:27 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > > I've generally recommended hpc.sourceforge as its simple, easy to
> > > install and has worked for me for a long time. [Uninstall requires a
> > > bit of effort though..]
> > >
> > > Sure - if one needs a bunch of packages including fortran -
> > > homebrew/macports would be the way to go.
> > >
> > > We've had quiet a few maint issues with macport conflicts - and Sean
> > > had been trying to resolve some of them within macpors. [And helping
> > > folks here on this list]
> > >
> > > Previously Homebrew had gfortran-4.2. But that was a buggy version and
> > > broke petsc f90 related functionality - so I didn't recommend it. But
> > > now I see gfortran-4.8 in homebrew - so perhaps it will work better
> > > now.
> > >
> > > Satish
> > >
> > > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> > >
> > > > Mark,
> > > >
> > > > You don't have any surprises in your configure file.  I'm not surprised
> > > > that your MacPorts install broke, we saw pretty terrible breakage across
> > > > the Scientific Python community, although I think Homebrew weathered the
> > > > update pretty well.
> > > >
> > > > I'd suggest following Sean's instructions so long as you're happy with
> > > Mac
> > > > Ports.  The most important thing is getting your compiler stack sane, and
> > > > unfortunately when you're compiling Fortran on OS X, you're going to have
> > > > to deal with a half-crazed stack no matter what you do.  See Geoff's
> > > > excellent summary on SciComp for future Fortran compiler options:
> > > > http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a reasonable
> > > > choice here.
> > > >
> > > > HashDist's main purpose is in helping scientists specify a software
> > > stack,
> > > > then reproduce it elsewhere.  It looks to me like PETSc is actually
> > > > satisfying most of your stack, and the only place where you need a little
> > > > help from MacPorts is the Fortran compiler, so I think HashDist would be
> > > > overkill for your needs here.
> > > >
> > > > Cheers,
> > > > Aron
> > > >
> > > >
> > > > On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > > >
> > > > > Sean, I seem to need to reinstall macorts.  I ran this:
> > > > >
> > > > > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix release)
> > > > > is now available:
> > > > >
> > > https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> > > > >
> > > > > And it created a MacPorts directory in Application but this just a few
> > > > > apps but no 'port' command.  Any idea what is going on here?
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > > > > sean.michael.farley at gmail.com> wrote:
> > > > >
> > > > >>
> > > > >> bsmith at mcs.anl.gov writes:
> > > > >>
> > > > >> >   I think resolved it by getting rid of some stuff that macports
> > > put in
> > > > >> maybe
> > > > >>
> > > > >> I just *completely* revamped the mpi ports in macports and would like
> > > to
> > > > >> know if these types of problems still exist.
> > > > >>
> > > > >> >   MPICH or libtool assumes certain files are there if other files
> > > are
> > > > >> there (without checking for them)
> > > > >> >
> > > > >> >    Barry
> > > > >> >
> > > > >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov>
> > > wrote:
> > > > >> >
> > > > >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> > > > >> >>
> > > > >> >>> Mark Adams <mfadams at lbl.gov> writes:
> > > > >> >>>
> > > > >> >>>> It seems to want /opt/local/lib/liblzma.la
> > > > >> >>>> I do have /opt/local/lib/liblzma.a
> > > > >> >>>
> > > > >> >>> There is no explicit reference to liblzma in either PETSc or
> > > MPICH.
> > > > >>  Can
> > > > >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> > > > >> >>
> > > > >> >> Ah - perhaps its a buggy libtool. Presumably its picked up from
> > > > >> >> /opt/local/bin/libtool - aka macports - and you have a broken
> > > macports
> > > > >> >> install.
> > > > >> >>
> > > > >> >> Satish
> > > > >> >>
> > > > >> >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > 
> 
> 



From mc0710 at gmail.com  Tue Jan 28 01:07:54 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 28 Jan 2014 01:07:54 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL (continued)
Message-ID: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>

Hi Karl,

I've been testing further, the code using TS with ViennaCL and there are a
couple of things I wanted to point out

1) When using the ComputeResidualViennaCL with either the normal Petsc
Vecs/Mats or Vec/MatViennaCL, and using the GPU, the nonlinear convergence
is very different from using an OpenCL CPU backend or just the regular
Petsc code.

a) Using NVIDIA OpenCL to run on the GPU to compute the residual and using
either normal Petsc Vec/Mat or ViennaCL Vec/Mat:

0 TS dt 10 time 0
    0 SNES Function norm 4.789374470711e-01
    1 SNES Function norm 5.491749197245e-02
    2 SNES Function norm 6.542412564158e-03
    3 SNES Function norm 7.800844032317e-04
    4 SNES Function norm 9.349243191537e-05
    5 SNES Function norm 1.120692741097e-05
1 TS dt 10 time 10

b) Using Intel OpenCL to run on the CPU to compute the residual and using
either normal Petsc Vec/Mat or ViennaCL Vec/Mat::

0 TS dt 10 time 0
    0 SNES Function norm 3.916582465172e-02
    1 SNES Function norm 4.990998832000e-07

c) Using ComputeResidual (which runs on the CPU) with the normal Petsc
Vec/Mat

0 TS dt 10 time 0
    0 SNES Function norm 3.916582465172e-02
    1 SNES Function norm 4.990998832000e-07
1 TS dt 10 time 10

You see that b) and c) match perfectly but a) is quite different. Why could
this be?

I also tried with the option -pc_type none cause I thought the GPU
preconditioners don't exist yet and I still get the above.

You can try with the attached code. I bumped up DOF to 8 and N1 and N2 to
128 each.
Run command:
./petsc_opencl -ts_monitor -snes_monitor -ts_max_steps 100 -ts_type theta
-ts_dt 10 -snes_rtol 1e-4 -ts_final_time 1000 -pc_type none

2) When I try using either ComputeResidual or ComputeResidualViennaCL with
the ViennaCL Vec/Mats, the GPU run crashes at a late time because of a
routine in ViennaCL.

ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
ViennaCL: Smaller work sizes could not solve the problem.
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
CL_MEM_OBJECT_ALLOCATION_FAILURE

I have attached the full crash log. The crash occurs late into the run, in
this case at the 80th time step. I thought all memory allocation occurs at
the beginning of the run, so I don't quite understand why its failing. Note
that the code works if I use ComputeResidualViennaCL with the normal Petsc
Vec/Mats.

Cheers,
Mani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/8f9f760d/attachment.html>
-------------- next part --------------
ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
ViennaCL: Smaller work sizes could not solve the problem. 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE 
 ViennaCL could not allocate memory on the device. Most likely the device simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at viennacl-support at lists.sourceforge.net and supply at least the following information:
 * Operating System
 * Which OpenCL implementation (AMD, NVIDIA, etc.)
 * ViennaCL version
Many thanks in advance!!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2368-g025cfdf  GIT Date: 2014-01-26 11:19:25 +0100
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc_opencl on a arch-linux2-c-debug named Deathstar by mc Tue Jan 28 00:24:44 2014
[0]PETSC ERROR: Libraries linked from /home/mc/Downloads/petsc_float_optimized/lib
[0]PETSC ERROR: Configure run at Mon Jan 27 17:37:55 2014
[0]PETSC ERROR: Configure options --prefix=/home/mc/Downloads/petsc_float_optimized/ --with-precision=single --with-debugging=0 COPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -qarch=native" --with-clean=1 --with-cuda=1 --with-opencl=1 --with-cuda-dir=/opt/cuda --download-txpetscgpu=yes --with-viennacl --download-viennacl make PETSC_DIR=/home/mc/Downloads/petsc PETSC_ARCH=arch-linux2-c-debug all
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatMult_SeqAIJViennaCL() line 196 in /home/mc/Downloads/petsc/src/mat/impls/aij/seq/seqviennacl/aijviennacl.cxx
[0]PETSC ERROR: MatMult() line 2242 in /home/mc/Downloads/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: PCApplyBAorAB() line 677 in /home/mc/Downloads/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApplyBAorAB() line 257 in /home/mc/Downloads/petsc/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPGMRESCycle() line 155 in /home/mc/Downloads/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 235 in /home/mc/Downloads/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 432 in /home/mc/Downloads/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 233 in /home/mc/Downloads/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3812 in /home/mc/Downloads/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_Theta() line 183 in /home/mc/Downloads/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: TSStep() line 2625 in /home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2741 in /home/mc/Downloads/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: main() line 90 in /home/mc/PhD/opencl_tests/petsc_opencl_fixed/petsc_opencl.cpp
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 76.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_opencl_fixed.tar.gz
Type: application/x-gzip
Size: 4978 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/8f9f760d/attachment.gz>

From goxberry at gmail.com  Tue Jan 28 01:39:06 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Mon, 27 Jan 2014 23:39:06 -0800
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401272344400.2447@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
Message-ID: <CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>

To echo what Aron said, I wouldn't point people at the
hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
it's a pain in the ass
to undo. The R/AT&T build of gcc was better, but also installed into
/usr/bin, and was also a pain in the ass to uninstall.

Having used both MacPorts (2010-2012) and Homebrew (2012-present), I find
Homebrew to be a better experience, especially if you only need a small
number of packages for development. MacPorts used to insist on its own
stack, which meant that if you wanted gfortran, you also had to install
many other packages.

I generally developed using gcc 4.2 because I found cross-version linking
to be a pain in the ass. I've also installed gcc 4.8 via the
homebrew/versions tap and that's worked well, too.

Python is sort of broken in both MacPorts and Homebrew. If you look at the
GitHub issues, there's been a lot of traffic related to Python in Homebrew
lately because they completely revamped how they handle Python in their
build recipes, which then broke some Python packages installed via
Homebrew. Last I checked, Python was more broken in MacPorts and required
lots of hacks to get things to work, but it's been a while since I've used
MacPorts. I think the best policy is to rely on the package manager for as
little Python software as possible, and install the rest of your Python
stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda sort of
does something similar, but I feel like conda is a great build system with
too many other responsibilities.


On Mon, Jan 27, 2014 at 9:53 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> BTW: gfortran from 'R' would install in /usr. It had an uninstaller
> [so was initially attractive to us].
>
> However if I remember correctly - it overwrote one of the libgcc_xx.a
> stuff that came with Xcode - and uninstalling it resulted in a broken
> gcc [I had to delete/reinstall xcode after that]
>
> There were other issues aswell. Perhaps Sean remembers them better..
> This was many osx/xcode versions ago.
>
> Satish
>
> On Mon, 27 Jan 2014, Satish Balay wrote:
>
> > Yeah - it must have been a while since I last checked on homebrew :)
> >
> > Wrt hpc.sourceforge.net - it installs in /usr/local - so it does not
> > conflict system stuff.
> >
> > Satish
> >
> > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> >
> > > Satish,
> > >
> > > Yup!  Samuel John switched homebrew over to a modern gfortran in 2012:
> > >
> > >
> https://github.com/Homebrew/homebrew/commit/1751362562f9b5d56708003f0dffb88e5b2418ab
> > >
> > > So it's been at least a year :)
> > >
> > > I don't point people at the hpc.sourceforge builds, if I recall
> correctly
> > > they install right into your /usr/bin, which seems a bit risky to me.
> > >
> > > Cheers,
> > > Aron
> > >
> > >
> > > On Tue, Jan 28, 2014 at 12:27 AM, Satish Balay <balay at mcs.anl.gov>
> wrote:
> > >
> > > > I've generally recommended hpc.sourceforge as its simple, easy to
> > > > install and has worked for me for a long time. [Uninstall requires a
> > > > bit of effort though..]
> > > >
> > > > Sure - if one needs a bunch of packages including fortran -
> > > > homebrew/macports would be the way to go.
> > > >
> > > > We've had quiet a few maint issues with macport conflicts - and Sean
> > > > had been trying to resolve some of them within macpors. [And helping
> > > > folks here on this list]
> > > >
> > > > Previously Homebrew had gfortran-4.2. But that was a buggy version
> and
> > > > broke petsc f90 related functionality - so I didn't recommend it. But
> > > > now I see gfortran-4.8 in homebrew - so perhaps it will work better
> > > > now.
> > > >
> > > > Satish
> > > >
> > > > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> > > >
> > > > > Mark,
> > > > >
> > > > > You don't have any surprises in your configure file.  I'm not
> surprised
> > > > > that your MacPorts install broke, we saw pretty terrible breakage
> across
> > > > > the Scientific Python community, although I think Homebrew
> weathered the
> > > > > update pretty well.
> > > > >
> > > > > I'd suggest following Sean's instructions so long as you're happy
> with
> > > > Mac
> > > > > Ports.  The most important thing is getting your compiler stack
> sane, and
> > > > > unfortunately when you're compiling Fortran on OS X, you're going
> to have
> > > > > to deal with a half-crazed stack no matter what you do.  See
> Geoff's
> > > > > excellent summary on SciComp for future Fortran compiler options:
> > > > > http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a
> reasonable
> > > > > choice here.
> > > > >
> > > > > HashDist's main purpose is in helping scientists specify a software
> > > > stack,
> > > > > then reproduce it elsewhere.  It looks to me like PETSc is actually
> > > > > satisfying most of your stack, and the only place where you need a
> little
> > > > > help from MacPorts is the Fortran compiler, so I think HashDist
> would be
> > > > > overkill for your needs here.
> > > > >
> > > > > Cheers,
> > > > > Aron
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov>
> wrote:
> > > > >
> > > > > > Sean, I seem to need to reinstall macorts.  I ran this:
> > > > > >
> > > > > > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix
> release)
> > > > > > is now available:
> > > > > >
> > > >
> https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> > > > > >
> > > > > > And it created a MacPorts directory in Application but this just
> a few
> > > > > > apps but no 'port' command.  Any idea what is going on here?
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > > > > > sean.michael.farley at gmail.com> wrote:
> > > > > >
> > > > > >>
> > > > > >> bsmith at mcs.anl.gov writes:
> > > > > >>
> > > > > >> >   I think resolved it by getting rid of some stuff that
> macports
> > > > put in
> > > > > >> maybe
> > > > > >>
> > > > > >> I just *completely* revamped the mpi ports in macports and
> would like
> > > > to
> > > > > >> know if these types of problems still exist.
> > > > > >>
> > > > > >> >   MPICH or libtool assumes certain files are there if other
> files
> > > > are
> > > > > >> there (without checking for them)
> > > > > >> >
> > > > > >> >    Barry
> > > > > >> >
> > > > > >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov
> >
> > > > wrote:
> > > > > >> >
> > > > > >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> > > > > >> >>
> > > > > >> >>> Mark Adams <mfadams at lbl.gov> writes:
> > > > > >> >>>
> > > > > >> >>>> It seems to want /opt/local/lib/liblzma.la
> > > > > >> >>>> I do have /opt/local/lib/liblzma.a
> > > > > >> >>>
> > > > > >> >>> There is no explicit reference to liblzma in either PETSc or
> > > > MPICH.
> > > > > >>  Can
> > > > > >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> > > > > >> >>
> > > > > >> >> Ah - perhaps its a buggy libtool. Presumably its picked up
> from
> > > > > >> >> /opt/local/bin/libtool - aka macports - and you have a broken
> > > > macports
> > > > > >> >> install.
> > > > > >> >>
> > > > > >> >> Satish
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> >
>
>


-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140127/acd360c3/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 28 03:09:42 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 28 Jan 2014 10:09:42 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
Message-ID: <52E773D6.8040407@mcs.anl.gov>

Hi Mani,

> I've been testing further, the code using TS with ViennaCL and there are
> a couple of things I wanted to point out
>
> 1) When using the ComputeResidualViennaCL with either the normal Petsc
> Vecs/Mats or Vec/MatViennaCL, and using the GPU, the nonlinear
> convergence is very different from using an OpenCL CPU backend or just
> the regular Petsc code.
>
> a) Using NVIDIA OpenCL to run on the GPU to compute the residual and
> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat:
>
> 0 TS dt 10 time 0
>      0 SNES Function norm 4.789374470711e-01
>      1 SNES Function norm 5.491749197245e-02
>      2 SNES Function norm 6.542412564158e-03
>      3 SNES Function norm 7.800844032317e-04
>      4 SNES Function norm 9.349243191537e-05
>      5 SNES Function norm 1.120692741097e-05
> 1 TS dt 10 time 10
>
> b) Using Intel OpenCL to run on the CPU to compute the residual and
> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat::
>
> 0 TS dt 10 time 0
>      0 SNES Function norm 3.916582465172e-02
>      1 SNES Function norm 4.990998832000e-07
>
> c) Using ComputeResidual (which runs on the CPU) with the normal Petsc
> Vec/Mat
>
> 0 TS dt 10 time 0
>      0 SNES Function norm 3.916582465172e-02
>      1 SNES Function norm 4.990998832000e-07
> 1 TS dt 10 time 10
>
> You see that b) and c) match perfectly but a) is quite different. Why
> could this be?

The reason are different arithmetic units. Your OpenCL kernel contains
dx_dt[INDEX_GLOBAL(i,j,var)] -
    (x[INDEX_GLOBAL(i+1,j,var)] -
     x[INDEX_GLOBAL(i,j,var)])/DX1 -
    (x[INDEX_GLOBAL(i,j+1,var)] -
    x[INDEX_GLOBAL(i,j,var)])/DX2
so you are subtracting values of about the same magnitude multiple 
times. You get consistent results on the CPU because the same arithmetic 
units get used irrespective of OpenCL-based or 'native' execution. The 
NVIDIA GPU has different round-off behavior. You are likely to see 
similar effects with AMD GPUs. There is nothing we can do to change this.


> 2) When I try using either ComputeResidual or ComputeResidualViennaCL
> with the ViennaCL Vec/Mats, the GPU run crashes at a late time because
> of a routine in ViennaCL.
>
> ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
> ViennaCL: Smaller work sizes could not solve the problem.
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
> CL_MEM_OBJECT_ALLOCATION_FAILURE
>
> I have attached the full crash log. The crash occurs late into the run,
> in this case at the 80th time step. I thought all memory allocation
> occurs at the beginning of the run, so I don't quite understand why its
> failing.

Okay, this sounds like the GPU ultimately runs out of memory. Which GPU 
do you use? How much memory does it have? Do you also see an increase in 
memory consumption with the Intel OpenCL SDK?


 > Note that the code works if I use ComputeResidualViennaCL with
 > the normal Petsc Vec/Mats.

You mean ComputeResidual(), don't you?

Best regards,
Karli



From balay at mcs.anl.gov  Tue Jan 28 08:39:16 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 08:39:16 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401280820250.2447@asterix>

On Tue, 28 Jan 2014, Geoff Oxberry wrote:

> To echo what Aron said, I wouldn't point people at the
> hpc.sourceforge.netbuilds. They do install directly into /usr/bin,
> and it's a pain in the ass to undo.

Just reiterating my previous e-mails. hpc installs in /usr/local/bin
and does not touch stuff in /usr/bin

> The R/AT&T build of gcc was better, but also installed into
> /usr/bin, and was also a pain in the ass to uninstall.

Our experience is R/AT&T build is worse wrt PETSc. Will never
recommend it. [and will never recommend gfortran-4.2 as this early
version has bugs in some of its f90 implementation affecting petsc's
f90 interface]

> Having used both MacPorts (2010-2012) and Homebrew (2012-present), I find
> Homebrew to be a better experience, especially if you only need a small
> number of packages for development. MacPorts used to insist on its own
> stack, which meant that if you wanted gfortran, you also had to install
> many other packages.
> 
> I generally developed using gcc 4.2 because I found cross-version linking
> to be a pain in the ass. I've also installed gcc 4.8 via the
> homebrew/versions tap and that's worked well, too.

One more reason to prefer hpc.sourceforge [gfortran only] install -
wrt PETSc.

> Python is sort of broken in both MacPorts and Homebrew. If you look at the
> GitHub issues, there's been a lot of traffic related to Python in Homebrew
> lately because they completely revamped how they handle Python in their
> build recipes, which then broke some Python packages installed via
> Homebrew. Last I checked, Python was more broken in MacPorts and required
> lots of hacks to get things to work, but it's been a while since I've used
> MacPorts. I think the best policy is to rely on the package manager for as
> little Python software as possible, and install the rest of your Python
> stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda sort of
> does something similar, but I feel like conda is a great build system with
> too many other responsibilities.

Sounds like you are adding more reasons to prefer hpc.sourceforge gfortran
[for petsc usage]

Obviously - if one needs more than just gfortran - they would have to
use Macports or homebrew - and deal with all the issues they
have. [Just like the issue Mark is currently having - hence this
e-mail thread]

I'll add one more note here: Its possible to cleanly uninstal hpc
stuff - but its not simple. One would have to keep the tarball arround
[or store the tarball table of contents in a file] - and use this
info to remove files.

cd /
sudo tar -tf ~/Downloads/gfortran-mlion.tar | xargs rm
sudo tar -tf ~/Downloads/gfortran-mlion.tar | xargs rmdir
sudo tar -tf ~/Downloads/gfortran-mlion.tar | xargs rmdir
...

Satish

> 
> On Mon, Jan 27, 2014 at 9:53 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > BTW: gfortran from 'R' would install in /usr. It had an uninstaller
> > [so was initially attractive to us].
> >
> > However if I remember correctly - it overwrote one of the libgcc_xx.a
> > stuff that came with Xcode - and uninstalling it resulted in a broken
> > gcc [I had to delete/reinstall xcode after that]
> >
> > There were other issues aswell. Perhaps Sean remembers them better..
> > This was many osx/xcode versions ago.
> >
> > Satish
> >
> > On Mon, 27 Jan 2014, Satish Balay wrote:
> >
> > > Yeah - it must have been a while since I last checked on homebrew :)
> > >
> > > Wrt hpc.sourceforge.net - it installs in /usr/local - so it does not
> > > conflict system stuff.
> > >
> > > Satish
> > >
> > > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> > >
> > > > Satish,
> > > >
> > > > Yup!  Samuel John switched homebrew over to a modern gfortran in 2012:
> > > >
> > > >
> > https://github.com/Homebrew/homebrew/commit/1751362562f9b5d56708003f0dffb88e5b2418ab
> > > >
> > > > So it's been at least a year :)
> > > >
> > > > I don't point people at the hpc.sourceforge builds, if I recall
> > correctly
> > > > they install right into your /usr/bin, which seems a bit risky to me.
> > > >
> > > > Cheers,
> > > > Aron
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 12:27 AM, Satish Balay <balay at mcs.anl.gov>
> > wrote:
> > > >
> > > > > I've generally recommended hpc.sourceforge as its simple, easy to
> > > > > install and has worked for me for a long time. [Uninstall requires a
> > > > > bit of effort though..]
> > > > >
> > > > > Sure - if one needs a bunch of packages including fortran -
> > > > > homebrew/macports would be the way to go.
> > > > >
> > > > > We've had quiet a few maint issues with macport conflicts - and Sean
> > > > > had been trying to resolve some of them within macpors. [And helping
> > > > > folks here on this list]
> > > > >
> > > > > Previously Homebrew had gfortran-4.2. But that was a buggy version
> > and
> > > > > broke petsc f90 related functionality - so I didn't recommend it. But
> > > > > now I see gfortran-4.8 in homebrew - so perhaps it will work better
> > > > > now.
> > > > >
> > > > > Satish
> > > > >
> > > > > On Mon, 27 Jan 2014, Aron Ahmadia wrote:
> > > > >
> > > > > > Mark,
> > > > > >
> > > > > > You don't have any surprises in your configure file.  I'm not
> > surprised
> > > > > > that your MacPorts install broke, we saw pretty terrible breakage
> > across
> > > > > > the Scientific Python community, although I think Homebrew
> > weathered the
> > > > > > update pretty well.
> > > > > >
> > > > > > I'd suggest following Sean's instructions so long as you're happy
> > with
> > > > > Mac
> > > > > > Ports.  The most important thing is getting your compiler stack
> > sane, and
> > > > > > unfortunately when you're compiling Fortran on OS X, you're going
> > to have
> > > > > > to deal with a half-crazed stack no matter what you do.  See
> > Geoff's
> > > > > > excellent summary on SciComp for future Fortran compiler options:
> > > > > > http://scicomp.stackexchange.com/a/2470/9 -- MacPorts is a
> > reasonable
> > > > > > choice here.
> > > > > >
> > > > > > HashDist's main purpose is in helping scientists specify a software
> > > > > stack,
> > > > > > then reproduce it elsewhere.  It looks to me like PETSc is actually
> > > > > > satisfying most of your stack, and the only place where you need a
> > little
> > > > > > help from MacPorts is the Fortran compiler, so I think HashDist
> > would be
> > > > > > overkill for your needs here.
> > > > > >
> > > > > > Cheers,
> > > > > > Aron
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 27, 2014 at 6:19 PM, Mark Adams <mfadams at lbl.gov>
> > wrote:
> > > > > >
> > > > > > > Sean, I seem to need to reinstall macorts.  I ran this:
> > > > > > >
> > > > > > > *Edit:* A binary installer for Mavericks (for the 2.2.1 bugfix
> > release)
> > > > > > > is now available:
> > > > > > >
> > > > >
> > https://distfiles.macports.org/MacPorts/MacPorts-2.2.1-10.9-Mavericks.pkg.
> > > > > > >
> > > > > > > And it created a MacPorts directory in Application but this just
> > a few
> > > > > > > apps but no 'port' command.  Any idea what is going on here?
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 27, 2014 at 11:44 AM, Sean Farley <
> > > > > > > sean.michael.farley at gmail.com> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> bsmith at mcs.anl.gov writes:
> > > > > > >>
> > > > > > >> >   I think resolved it by getting rid of some stuff that
> > macports
> > > > > put in
> > > > > > >> maybe
> > > > > > >>
> > > > > > >> I just *completely* revamped the mpi ports in macports and
> > would like
> > > > > to
> > > > > > >> know if these types of problems still exist.
> > > > > > >>
> > > > > > >> >   MPICH or libtool assumes certain files are there if other
> > files
> > > > > are
> > > > > > >> there (without checking for them)
> > > > > > >> >
> > > > > > >> >    Barry
> > > > > > >> >
> > > > > > >> > On Jan 27, 2014, at 10:36 AM, Satish Balay <balay at mcs.anl.gov
> > >
> > > > > wrote:
> > > > > > >> >
> > > > > > >> >> On Mon, 27 Jan 2014, Jed Brown wrote:
> > > > > > >> >>
> > > > > > >> >>> Mark Adams <mfadams at lbl.gov> writes:
> > > > > > >> >>>
> > > > > > >> >>>> It seems to want /opt/local/lib/liblzma.la
> > > > > > >> >>>> I do have /opt/local/lib/liblzma.a
> > > > > > >> >>>
> > > > > > >> >>> There is no explicit reference to liblzma in either PETSc or
> > > > > MPICH.
> > > > > > >>  Can
> > > > > > >> >>> you send PETSC_ARCH/externalpackages/mpich*/config.log?
> > > > > > >> >>
> > > > > > >> >> Ah - perhaps its a buggy libtool. Presumably its picked up
> > from
> > > > > > >> >> /opt/local/bin/libtool - aka macports - and you have a broken
> > > > > macports
> > > > > > >> >> install.
> > > > > > >> >>
> > > > > > >> >> Satish
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
> 
> 
> 



From michael.lange at imperial.ac.uk  Tue Jan 28 08:57:43 2014
From: michael.lange at imperial.ac.uk (Michael Lange)
Date: Tue, 28 Jan 2014 14:57:43 +0000
Subject: [petsc-dev] Cell overlap in DMPlexDistribute
Message-ID: <52E7C567.7050101@imperial.ac.uk>

Hi,

I noticed that the cell overlap created during DMPlexDistribute does not 
include cells that only share a vertex but no edge with an owned cell. 
This causes problems when performing local assembly 
(MAT_IGNORE_OFF_PROC_ENTRIES) based on information from the plex, 
because one contribution to the shared vertex is missing.

As an example consider the 2x2 square with 8 cells (attached). When 
partitioned across two ranks with overlap=1 each rank owns 4 cells and 
in the current version knows about 2 halo cells, giving a total of 6 
cells. The central vertex, however, touches 3 owned and 3 non-owned 
cells, one of which doesn't share an edge with any owned cells. So, in 
order to correctly assemble the central vertex locally, the rank owning 
it needs to know about 7 cells in total.

I have attached a patch that fixes this problem by going over the 
inverse closure of all vertices associated with a given cell instead of 
using the provided edge graph. Please tell me what you think and whether 
there might be an easier way to fix this.

Kind regards
Michael Lange
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix-dmplex-cell-overlap.patch
Type: text/x-patch
Size: 3382 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/735c9383/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: plex_square.c
Type: text/x-csrc
Size: 1335 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/735c9383/attachment-0001.bin>

From mc0710 at gmail.com  Tue Jan 28 09:56:14 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 28 Jan 2014 09:56:14 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <52E773D6.8040407@mcs.anl.gov>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
	<52E773D6.8040407@mcs.anl.gov>
Message-ID: <CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>

Hi Karl,

Thanks for the explanation. Do you think it will help if I use a GPU which
is capable of doing double precision arithmetic?

I am using NVIDIA Quadra FX 1800 M. It has 1GB of global memory.
Unfortunately NVIDIAs' visual profiler does not seem to work with its
OpenCL implementation. The code does not crash when I run it on the CPU
using Intels OpenCL.

I mean't to say that the code does not crash either with
ComputeResidualViennaCL or ComputeResidual with the normal Petsc Vec/Mats
but does indeed crash when either of them are used with the ViennaCL vecs.
Do you think there is memory allocation at every time step? I thought all
the memory would be allocated during initialization.

Cheers,
Mani




On Tue, Jan 28, 2014 at 3:09 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
>  I've been testing further, the code using TS with ViennaCL and there are
>> a couple of things I wanted to point out
>>
>> 1) When using the ComputeResidualViennaCL with either the normal Petsc
>> Vecs/Mats or Vec/MatViennaCL, and using the GPU, the nonlinear
>> convergence is very different from using an OpenCL CPU backend or just
>> the regular Petsc code.
>>
>> a) Using NVIDIA OpenCL to run on the GPU to compute the residual and
>> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat:
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 4.789374470711e-01
>>      1 SNES Function norm 5.491749197245e-02
>>      2 SNES Function norm 6.542412564158e-03
>>      3 SNES Function norm 7.800844032317e-04
>>      4 SNES Function norm 9.349243191537e-05
>>      5 SNES Function norm 1.120692741097e-05
>> 1 TS dt 10 time 10
>>
>> b) Using Intel OpenCL to run on the CPU to compute the residual and
>> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat::
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 3.916582465172e-02
>>      1 SNES Function norm 4.990998832000e-07
>>
>> c) Using ComputeResidual (which runs on the CPU) with the normal Petsc
>> Vec/Mat
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 3.916582465172e-02
>>      1 SNES Function norm 4.990998832000e-07
>> 1 TS dt 10 time 10
>>
>> You see that b) and c) match perfectly but a) is quite different. Why
>> could this be?
>>
>
> The reason are different arithmetic units. Your OpenCL kernel contains
> dx_dt[INDEX_GLOBAL(i,j,var)] -
>    (x[INDEX_GLOBAL(i+1,j,var)] -
>     x[INDEX_GLOBAL(i,j,var)])/DX1 -
>    (x[INDEX_GLOBAL(i,j+1,var)] -
>    x[INDEX_GLOBAL(i,j,var)])/DX2
> so you are subtracting values of about the same magnitude multiple times.
> You get consistent results on the CPU because the same arithmetic units get
> used irrespective of OpenCL-based or 'native' execution. The NVIDIA GPU has
> different round-off behavior. You are likely to see similar effects with
> AMD GPUs. There is nothing we can do to change this.
>
>
>
>  2) When I try using either ComputeResidual or ComputeResidualViennaCL
>> with the ViennaCL Vec/Mats, the GPU run crashes at a late time because
>> of a routine in ViennaCL.
>>
>> ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
>> ViennaCL: Smaller work sizes could not solve the problem.
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Error in external library!
>> [0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
>> CL_MEM_OBJECT_ALLOCATION_FAILURE
>>
>> I have attached the full crash log. The crash occurs late into the run,
>> in this case at the 80th time step. I thought all memory allocation
>> occurs at the beginning of the run, so I don't quite understand why its
>> failing.
>>
>
> Okay, this sounds like the GPU ultimately runs out of memory. Which GPU do
> you use? How much memory does it have? Do you also see an increase in
> memory consumption with the Intel OpenCL SDK?
>
>
>
> > Note that the code works if I use ComputeResidualViennaCL with
> > the normal Petsc Vec/Mats.
>
> You mean ComputeResidual(), don't you?
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/85a9b5cf/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 28 10:12:59 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 28 Jan 2014 17:12:59 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>	<52E773D6.8040407@mcs.anl.gov>
	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>
Message-ID: <52E7D70B.3050107@mcs.anl.gov>

Hi Mani,

 > Thanks for the explanation. Do you think it will help if I use a GPU
> which is capable of doing double precision arithmetic?

Your GPU must be supporting double precision, otherwise the jit-compiler 
will fail. However, your GPU might emulate double precision arithmetics 
poorly.

> I am using NVIDIA Quadra FX 1800 M. It has 1GB of global memory.

For production runs you definitely want to use a discrete GPU to get the 
benefits of higher bandwidth (through higher heat dissipation...)


> Unfortunately NVIDIAs' visual profiler does not seem to work with its
> OpenCL implementation. The code does not crash when I run it on the CPU
> using Intels OpenCL.

Let me put it this way: This ridiculous move of taking OpenCL debugging 
capabilities out when releasing CUDA 5.0 is not based on scientific 
reasoning...


> I mean't to say that the code does not crash either with
> ComputeResidualViennaCL or ComputeResidual with the normal Petsc
> Vec/Mats but does indeed crash when either of them are used with the
> ViennaCL vecs.

I'm wondering how ComputeResidualViennaCL will work if the vector type 
is not AIJVIENNACL?

> Do you think there is memory allocation at every time
> step? I thought all the memory would be allocated during initialization.

 From your description a memory allocation seems to be the case, yes, 
but I still need to verify this. Did you see a similar increase in 
memory consumption (use e.g. top) with Intel's OpenCL SDK?

Best regards,
Karli




From sean.michael.farley at gmail.com  Tue Jan 28 11:10:35 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 11:10:35 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
Message-ID: <m2lhy02fsk.fsf@gmail.com>


goxberry at gmail.com writes:

> To echo what Aron said, I wouldn't point people at the
> hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
> it's a pain in the ass
> to undo. The R/AT&T build of gcc was better, but also installed into
> /usr/bin, and was also a pain in the ass to uninstall.

It's impossible to uninstall since it overwrites gcc binaries. So you
have to reinstall Xcode / Command Line Tools to go back.

The biggest reason I don't recommend the gfortran binary from R or from
hpc.sourceforge.net is that it puts you in a spot of maintaining your
own stack. Now that I've integrated all my custom modifications into
MacPorts-proper, I can finally start pointing people to that.

That's not to say that I think MacPorts is the best solution for
everyone. I just found that I could get MacPorts to do what I wanted
with the least amount of work.

> Having used both MacPorts (2010-2012) and Homebrew (2012-present), I find
> Homebrew to be a better experience, especially if you only need a small
> number of packages for development. MacPorts used to insist on its own
> stack, which meant that if you wanted gfortran, you also had to install
> many other packages.

MacPorts still does this so that it can get a sane stack that is
invariant of OS versions and processor type (we still support ppc
:-(). A year or two ago we got buildbots so that most people can benefit
from binary downloads.

Over the last year, I've finally been able to become maintainer for a
lot (~100) of the scientific packages in MacPorts (including PETSc and
friends) with the goal of improving the 

> I generally developed using gcc 4.2 because I found cross-version linking
> to be a pain in the ass. I've also installed gcc 4.8 via the
> homebrew/versions tap and that's worked well, too.

My problem with homebrew's use of compilers is that I couldn't specify
(easily) a way to install a package with either compiler and then switch
between the two packages (a la PETSC_ARCH).

> Python is sort of broken in both MacPorts and Homebrew. If you look at the
> GitHub issues, there's been a lot of traffic related to Python in Homebrew
> lately because they completely revamped how they handle Python in their
> build recipes, which then broke some Python packages installed via
> Homebrew. Last I checked, Python was more broken in MacPorts and required
> lots of hacks to get things to work, but it's been a while since I've used
> MacPorts. I think the best policy is to rely on the package manager for as
> little Python software as possible, and install the rest of your Python
> stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda sort of
> does something similar, but I feel like conda is a great build system with
> too many other responsibilities.

Woah, woah, woah there. Python is broken? I've been using it in MacPorts
for years with no issues. If you can reproduce anything or point me to
the tickets on trac.macports.org, I can try to fix them.


From mc0710 at gmail.com  Tue Jan 28 11:49:49 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 28 Jan 2014 11:49:49 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <52E7D70B.3050107@mcs.anl.gov>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
	<52E773D6.8040407@mcs.anl.gov>
	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>
	<52E7D70B.3050107@mcs.anl.gov>
Message-ID: <CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>

Hi Karl,

It actually works even if I use ComputeResidualViennaCL without using
ViennaCL vecs! Isn't it because there is a consistency check to see where
the data resides whenever there is a call to VecViennaCLGetArray?

I reran the code on another machine with a NVIDIA GeForce 550 ti with 1GB
of global memory. It still fails with the following error:

ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
ViennaCL: Smaller work sizes could not solve the problem.
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
CL_MEM_OBJECT_ALLOCATION_FAILURE
 ViennaCL could not allocate memory on the device. Most likely the device
simply ran out of memory.

To reproduce the error, I only had to turn on the ViennaCL vecs. The error
occurs irrespective of which residual evaluation function I use. I checked
to see if there was any memory usage increase in the CPU memory while the
run was going on but there was none.

Cheers,
Mani


On Tue, Jan 28, 2014 at 10:12 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > Thanks for the explanation. Do you think it will help if I use a GPU
>
>> which is capable of doing double precision arithmetic?
>>
>
> Your GPU must be supporting double precision, otherwise the jit-compiler
> will fail. However, your GPU might emulate double precision arithmetics
> poorly.
>
>
>  I am using NVIDIA Quadra FX 1800 M. It has 1GB of global memory.
>>
>
> For production runs you definitely want to use a discrete GPU to get the
> benefits of higher bandwidth (through higher heat dissipation...)
>
>
>
>  Unfortunately NVIDIAs' visual profiler does not seem to work with its
>> OpenCL implementation. The code does not crash when I run it on the CPU
>> using Intels OpenCL.
>>
>
> Let me put it this way: This ridiculous move of taking OpenCL debugging
> capabilities out when releasing CUDA 5.0 is not based on scientific
> reasoning...
>
>
>
>  I mean't to say that the code does not crash either with
>> ComputeResidualViennaCL or ComputeResidual with the normal Petsc
>> Vec/Mats but does indeed crash when either of them are used with the
>> ViennaCL vecs.
>>
>
> I'm wondering how ComputeResidualViennaCL will work if the vector type is
> not AIJVIENNACL?
>
>
>  Do you think there is memory allocation at every time
>> step? I thought all the memory would be allocated during initialization.
>>
>
> From your description a memory allocation seems to be the case, yes, but I
> still need to verify this. Did you see a similar increase in memory
> consumption (use e.g. top) with Intel's OpenCL SDK?
>
> Best regards,
> Karli
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/18047623/attachment.html>

From mc0710 at gmail.com  Tue Jan 28 11:51:33 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 28 Jan 2014 11:51:33 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
	<52E773D6.8040407@mcs.anl.gov>
	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>
	<52E7D70B.3050107@mcs.anl.gov>
	<CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>
Message-ID: <CAJzU6sAQ+jSebyp5G1uk9+oqMDMCHHSX7x9pU7fuWupEf9-Quw@mail.gmail.com>

Note that the error occurs in the middle of the run, after a certain number
of time steps. The time step at which it fails is different for the two
machines for which I tested. And it only happens when I use the GPU. The
OpenCL backend using a CPU with Intel opencl works fine.

Cheers,
Mani


On Tue, Jan 28, 2014 at 11:49 AM, Mani Chandra <mc0710 at gmail.com> wrote:

> Hi Karl,
>
> It actually works even if I use ComputeResidualViennaCL without using
> ViennaCL vecs! Isn't it because there is a consistency check to see where
> the data resides whenever there is a call to VecViennaCLGetArray?
>
> I reran the code on another machine with a NVIDIA GeForce 550 ti with 1GB
> of global memory. It still fails with the following error:
>
> ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
> ViennaCL: Smaller work sizes could not solve the problem.
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
> CL_MEM_OBJECT_ALLOCATION_FAILURE
>  ViennaCL could not allocate memory on the device. Most likely the device
> simply ran out of memory.
>
> To reproduce the error, I only had to turn on the ViennaCL vecs. The error
> occurs irrespective of which residual evaluation function I use. I checked
> to see if there was any memory usage increase in the CPU memory while the
> run was going on but there was none.
>
> Cheers,
> Mani
>
>
> On Tue, Jan 28, 2014 at 10:12 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:
>
>> Hi Mani,
>>
>>
>> > Thanks for the explanation. Do you think it will help if I use a GPU
>>
>>> which is capable of doing double precision arithmetic?
>>>
>>
>> Your GPU must be supporting double precision, otherwise the jit-compiler
>> will fail. However, your GPU might emulate double precision arithmetics
>> poorly.
>>
>>
>>  I am using NVIDIA Quadra FX 1800 M. It has 1GB of global memory.
>>>
>>
>> For production runs you definitely want to use a discrete GPU to get the
>> benefits of higher bandwidth (through higher heat dissipation...)
>>
>>
>>
>>  Unfortunately NVIDIAs' visual profiler does not seem to work with its
>>> OpenCL implementation. The code does not crash when I run it on the CPU
>>> using Intels OpenCL.
>>>
>>
>> Let me put it this way: This ridiculous move of taking OpenCL debugging
>> capabilities out when releasing CUDA 5.0 is not based on scientific
>> reasoning...
>>
>>
>>
>>  I mean't to say that the code does not crash either with
>>> ComputeResidualViennaCL or ComputeResidual with the normal Petsc
>>> Vec/Mats but does indeed crash when either of them are used with the
>>> ViennaCL vecs.
>>>
>>
>> I'm wondering how ComputeResidualViennaCL will work if the vector type is
>> not AIJVIENNACL?
>>
>>
>>  Do you think there is memory allocation at every time
>>> step? I thought all the memory would be allocated during initialization.
>>>
>>
>> From your description a memory allocation seems to be the case, yes, but
>> I still need to verify this. Did you see a similar increase in memory
>> consumption (use e.g. top) with Intel's OpenCL SDK?
>>
>> Best regards,
>> Karli
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/05107190/attachment.html>

From goxberry at gmail.com  Tue Jan 28 12:14:21 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Tue, 28 Jan 2014 10:14:21 -0800
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2lhy02fsk.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
Message-ID: <CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>

On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley
<sean.michael.farley at gmail.com>wrote:

>
> goxberry at gmail.com writes:
>
> > To echo what Aron said, I wouldn't point people at the
> > hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
> > it's a pain in the ass
>

Satish is probably right here about the build location. It's been three or
four years since I've installed it this way. I stand by that it's still
difficult to revert. I actually tried this method because of PETSc and
regretted it because the experience was terrible. Using a package manager
is more maintainable, and I think PETSc's recommendation of the
hpc.sourceforge build is a disservice to both users and to PETSc's
excellent reputation.


> > to undo. The R/AT&T build of gcc was better, but also installed into
> > /usr/bin, and was also a pain in the ass to uninstall.
>
> It's impossible to uninstall since it overwrites gcc binaries. So you
> have to reinstall Xcode / Command Line Tools to go back.
>

Functionally, that's the same thing. Again, it's been three or four years.


>
> The biggest reason I don't recommend the gfortran binary from R or from
> hpc.sourceforge.net is that it puts you in a spot of maintaining your
> own stack. Now that I've integrated all my custom modifications into
> MacPorts-proper, I can finally start pointing people to that.
>
> That's not to say that I think MacPorts is the best solution for
> everyone. I just found that I could get MacPorts to do what I wanted
> with the least amount of work.


That was not my experience.


> > Having used both MacPorts (2010-2012) and Homebrew (2012-present), I find
> > Homebrew to be a better experience, especially if you only need a small
> > number of packages for development. MacPorts used to insist on its own
> > stack, which meant that if you wanted gfortran, you also had to install
> > many other packages.
>
> MacPorts still does this so that it can get a sane stack that is
> invariant of OS versions and processor type (we still support ppc
> :-(). A year or two ago we got buildbots so that most people can benefit
> from binary downloads.
>

Homebrew and MacPorts use different philosophies. Homebrew relies more on
the system stack, which led to all sorts of problems when Mavericks came
out. Most of these seemed to be due to the clang transition to libc++, or
to C++11. Mike McQuaid, Misty DeMeo, Jack Nagel, and Adam Vandenburg have
been doing an excellent job fixing bugs.


> Over the last year, I've finally been able to become maintainer for a
> lot (~100) of the scientific packages in MacPorts (including PETSc and
> friends) with the goal of improving the


I used to follow that. It seemed like it took a long time for MacPorts to
respond to bug reports, and they weren't particularly keen on external
contributions. I haven't had that issue with Homebrew; all their core devs
are very engaged.


> > I generally developed using gcc 4.2 because I found cross-version linking
> > to be a pain in the ass. I've also installed gcc 4.8 via the
> > homebrew/versions tap and that's worked well, too.
>
> My problem with homebrew's use of compilers is that I couldn't specify
> (easily) a way to install a package with either compiler and then switch
> between the two packages (a la PETSC_ARCH).


You're right; MacPorts does that better. There may be a way to do that with
flags. Some formulas in Homebrew build with gcc instead of clang. At worst,
you could maintain your own taps of formulas that you want to compile with
specific compilers. It would be an interesting idea to pitch.


> > Python is sort of broken in both MacPorts and Homebrew. If you look at
> the
> > GitHub issues, there's been a lot of traffic related to Python in
> Homebrew
> > lately because they completely revamped how they handle Python in their
> > build recipes, which then broke some Python packages installed via
> > Homebrew. Last I checked, Python was more broken in MacPorts and required
> > lots of hacks to get things to work, but it's been a while since I've
> used
> > MacPorts. I think the best policy is to rely on the package manager for
> as
> > little Python software as possible, and install the rest of your Python
> > stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda sort
> of
> > does something similar, but I feel like conda is a great build system
> with
> > too many other responsibilities.
>
> Woah, woah, woah there. Python is broken? I've been using it in MacPorts
> for years with no issues. If you can reproduce anything or point me to
> the tickets on trac.macports.org, I can try to fix them.
>

Aron already went over this in his comments on the Scientific Python stack.

I've basically stopped using MacPorts as of two years ago because I found
that to make things work, I had to repeatedly kludge around things, making
building software from scratch quite painful. I think I submitted maybe one
or two tickets to MacPorts, and didn't get much of a response. I don't plan
on resurrecting my old install just to file bug reports. I switched to
Homebrew because the UX in MacPorts was getting miserable, it's written in
Ruby (not Tcl), they're much more welcoming about contributions (submitting
a PR was painless), and there's much more of a community there. I don't
need to kludge things, and they do an excellent job of responding to bugs.

I applaud what MacPorts is doing, and as a former user, my biggest
suggestion would be to make it a little more clear that MacPorts values its
user community. There are some bug reports that go unresolved for a year or
more, which makes it hard to stick with MacPorts if you're one of the users
that keeps dealing with that bug. MacPorts also gives off the impression
that it's hard to contribute (Homebrew has 3450 contributors according to
GitHub, and is GitHub's most starred/forked repo; MacPorts has 182
contributors according to Ohloh). There are a lot of similar complaints in
"Homebrew vs MacPorts" posts, which is why Homebrew seems to be eating
MacPorts' lunch, despite MacPorts' big head start, and initially far
superior feature set.

-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/c077e230/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan 28 12:22:05 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 28 Jan 2014 12:22:05 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
Message-ID: <B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>


On Jan 28, 2014, at 12:14 PM, Geoff Oxberry <goxberry at gmail.com> wrote:

> 
> 
> 
> On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
> 
> goxberry at gmail.com writes:
> 
> > To echo what Aron said, I wouldn't point people at the
> > hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
> > it's a pain in the ass
> 
> Satish is probably right here about the build location. It's been three or four years since I've installed it this way. I stand by that it's still difficult to revert. I actually tried this method because of PETSc and regretted it because the experience was terrible. Using a package manager is more maintainable, and I think PETSc's recommendation of the hpc.sourceforge build is a disservice to both users and to PETSc's excellent reputation.

   I think package managers for Mac OS are a disservice to the community and recommend not using them. (See all the discussions in these emails about how they fuck up).

   Barry

>  
> > to undo. The R/AT&T build of gcc was better, but also installed into
> > /usr/bin, and was also a pain in the ass to uninstall.
> 
> It's impossible to uninstall since it overwrites gcc binaries. So you
> have to reinstall Xcode / Command Line Tools to go back.
> 
> Functionally, that's the same thing. Again, it's been three or four years.
>  
> 
> The biggest reason I don't recommend the gfortran binary from R or from
> hpc.sourceforge.net is that it puts you in a spot of maintaining your
> own stack. Now that I've integrated all my custom modifications into
> MacPorts-proper, I can finally start pointing people to that.
> 
> That's not to say that I think MacPorts is the best solution for
> everyone. I just found that I could get MacPorts to do what I wanted
> with the least amount of work.
> 
> That was not my experience. 
>  
> > Having used both MacPorts (2010-2012) and Homebrew (2012-present), I find
> > Homebrew to be a better experience, especially if you only need a small
> > number of packages for development. MacPorts used to insist on its own
> > stack, which meant that if you wanted gfortran, you also had to install
> > many other packages.
> 
> MacPorts still does this so that it can get a sane stack that is
> invariant of OS versions and processor type (we still support ppc
> :-(). A year or two ago we got buildbots so that most people can benefit
> from binary downloads.
> 
> Homebrew and MacPorts use different philosophies. Homebrew relies more on the system stack, which led to all sorts of problems when Mavericks came out. Most of these seemed to be due to the clang transition to libc++, or to C++11. Mike McQuaid, Misty DeMeo, Jack Nagel, and Adam Vandenburg have been doing an excellent job fixing bugs.
>  
> Over the last year, I've finally been able to become maintainer for a
> lot (~100) of the scientific packages in MacPorts (including PETSc and
> friends) with the goal of improving the
> 
> I used to follow that. It seemed like it took a long time for MacPorts to respond to bug reports, and they weren't particularly keen on external contributions. I haven't had that issue with Homebrew; all their core devs are very engaged. 
>  
> > I generally developed using gcc 4.2 because I found cross-version linking
> > to be a pain in the ass. I've also installed gcc 4.8 via the
> > homebrew/versions tap and that's worked well, too.
> 
> My problem with homebrew's use of compilers is that I couldn't specify
> (easily) a way to install a package with either compiler and then switch
> between the two packages (a la PETSC_ARCH).
> 
> You're right; MacPorts does that better. There may be a way to do that with flags. Some formulas in Homebrew build with gcc instead of clang. At worst, you could maintain your own taps of formulas that you want to compile with specific compilers. It would be an interesting idea to pitch.
>  
> > Python is sort of broken in both MacPorts and Homebrew. If you look at the
> > GitHub issues, there's been a lot of traffic related to Python in Homebrew
> > lately because they completely revamped how they handle Python in their
> > build recipes, which then broke some Python packages installed via
> > Homebrew. Last I checked, Python was more broken in MacPorts and required
> > lots of hacks to get things to work, but it's been a while since I've used
> > MacPorts. I think the best policy is to rely on the package manager for as
> > little Python software as possible, and install the rest of your Python
> > stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda sort of
> > does something similar, but I feel like conda is a great build system with
> > too many other responsibilities.
> 
> Woah, woah, woah there. Python is broken? I've been using it in MacPorts
> for years with no issues. If you can reproduce anything or point me to
> the tickets on trac.macports.org, I can try to fix them.
> 
> Aron already went over this in his comments on the Scientific Python stack.
> 
> I've basically stopped using MacPorts as of two years ago because I found that to make things work, I had to repeatedly kludge around things, making building software from scratch quite painful. I think I submitted maybe one or two tickets to MacPorts, and didn't get much of a response. I don't plan on resurrecting my old install just to file bug reports. I switched to Homebrew because the UX in MacPorts was getting miserable, it's written in Ruby (not Tcl), they're much more welcoming about contributions (submitting a PR was painless), and there's much more of a community there. I don't need to kludge things, and they do an excellent job of responding to bugs.
> 
> I applaud what MacPorts is doing, and as a former user, my biggest suggestion would be to make it a little more clear that MacPorts values its user community. There are some bug reports that go unresolved for a year or more, which makes it hard to stick with MacPorts if you're one of the users that keeps dealing with that bug. MacPorts also gives off the impression that it's hard to contribute (Homebrew has 3450 contributors according to GitHub, and is GitHub's most starred/forked repo; MacPorts has 182 contributors according to Ohloh). There are a lot of similar complaints in "Homebrew vs MacPorts" posts, which is why Homebrew seems to be eating MacPorts' lunch, despite MacPorts' big head start, and initially far superior feature set.
> 
> -- 
> Geoffrey Oxberry, Ph.D., E.I.T.
> goxberry at gmail.com



From knepley at gmail.com  Tue Jan 28 12:23:15 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 12:23:15 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
Message-ID: <CAMYG4Gk4=RZWgUFoH3LFx5WePYtjoYeZezdF9J2rno=GeR1cEw@mail.gmail.com>

On Tue, Jan 28, 2014 at 12:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Jan 28, 2014, at 12:14 PM, Geoff Oxberry <goxberry at gmail.com> wrote:
>
> >
> >
> >
> > On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley <
> sean.michael.farley at gmail.com> wrote:
> >
> > goxberry at gmail.com writes:
> >
> > > To echo what Aron said, I wouldn't point people at the
> > > hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
> > > it's a pain in the ass
> >
> > Satish is probably right here about the build location. It's been three
> or four years since I've installed it this way. I stand by that it's still
> difficult to revert. I actually tried this method because of PETSc and
> regretted it because the experience was terrible. Using a package manager
> is more maintainable, and I think PETSc's recommendation of the
> hpc.sourceforge build is a disservice to both users and to PETSc's
> excellent reputation.
>
>    I think package managers for Mac OS are a disservice to the community
> and recommend not using them. (See all the discussions in these emails
> about how they fuck up).


+1 I use MattPorts.

   Matt


>
>    Barry
>
> >
> > > to undo. The R/AT&T build of gcc was better, but also installed into
> > > /usr/bin, and was also a pain in the ass to uninstall.
> >
> > It's impossible to uninstall since it overwrites gcc binaries. So you
> > have to reinstall Xcode / Command Line Tools to go back.
> >
> > Functionally, that's the same thing. Again, it's been three or four
> years.
> >
> >
> > The biggest reason I don't recommend the gfortran binary from R or from
> > hpc.sourceforge.net is that it puts you in a spot of maintaining your
> > own stack. Now that I've integrated all my custom modifications into
> > MacPorts-proper, I can finally start pointing people to that.
> >
> > That's not to say that I think MacPorts is the best solution for
> > everyone. I just found that I could get MacPorts to do what I wanted
> > with the least amount of work.
> >
> > That was not my experience.
> >
> > > Having used both MacPorts (2010-2012) and Homebrew (2012-present), I
> find
> > > Homebrew to be a better experience, especially if you only need a small
> > > number of packages for development. MacPorts used to insist on its own
> > > stack, which meant that if you wanted gfortran, you also had to install
> > > many other packages.
> >
> > MacPorts still does this so that it can get a sane stack that is
> > invariant of OS versions and processor type (we still support ppc
> > :-(). A year or two ago we got buildbots so that most people can benefit
> > from binary downloads.
> >
> > Homebrew and MacPorts use different philosophies. Homebrew relies more
> on the system stack, which led to all sorts of problems when Mavericks came
> out. Most of these seemed to be due to the clang transition to libc++, or
> to C++11. Mike McQuaid, Misty DeMeo, Jack Nagel, and Adam Vandenburg have
> been doing an excellent job fixing bugs.
> >
> > Over the last year, I've finally been able to become maintainer for a
> > lot (~100) of the scientific packages in MacPorts (including PETSc and
> > friends) with the goal of improving the
> >
> > I used to follow that. It seemed like it took a long time for MacPorts
> to respond to bug reports, and they weren't particularly keen on external
> contributions. I haven't had that issue with Homebrew; all their core devs
> are very engaged.
> >
> > > I generally developed using gcc 4.2 because I found cross-version
> linking
> > > to be a pain in the ass. I've also installed gcc 4.8 via the
> > > homebrew/versions tap and that's worked well, too.
> >
> > My problem with homebrew's use of compilers is that I couldn't specify
> > (easily) a way to install a package with either compiler and then switch
> > between the two packages (a la PETSC_ARCH).
> >
> > You're right; MacPorts does that better. There may be a way to do that
> with flags. Some formulas in Homebrew build with gcc instead of clang. At
> worst, you could maintain your own taps of formulas that you want to
> compile with specific compilers. It would be an interesting idea to pitch.
> >
> > > Python is sort of broken in both MacPorts and Homebrew. If you look at
> the
> > > GitHub issues, there's been a lot of traffic related to Python in
> Homebrew
> > > lately because they completely revamped how they handle Python in their
> > > build recipes, which then broke some Python packages installed via
> > > Homebrew. Last I checked, Python was more broken in MacPorts and
> required
> > > lots of hacks to get things to work, but it's been a while since I've
> used
> > > MacPorts. I think the best policy is to rely on the package manager
> for as
> > > little Python software as possible, and install the rest of your Python
> > > stack in an isolated manner; I use pyenv, pip, & virtualenv. Conda
> sort of
> > > does something similar, but I feel like conda is a great build system
> with
> > > too many other responsibilities.
> >
> > Woah, woah, woah there. Python is broken? I've been using it in MacPorts
> > for years with no issues. If you can reproduce anything or point me to
> > the tickets on trac.macports.org, I can try to fix them.
> >
> > Aron already went over this in his comments on the Scientific Python
> stack.
> >
> > I've basically stopped using MacPorts as of two years ago because I
> found that to make things work, I had to repeatedly kludge around things,
> making building software from scratch quite painful. I think I submitted
> maybe one or two tickets to MacPorts, and didn't get much of a response. I
> don't plan on resurrecting my old install just to file bug reports. I
> switched to Homebrew because the UX in MacPorts was getting miserable, it's
> written in Ruby (not Tcl), they're much more welcoming about contributions
> (submitting a PR was painless), and there's much more of a community there.
> I don't need to kludge things, and they do an excellent job of responding
> to bugs.
> >
> > I applaud what MacPorts is doing, and as a former user, my biggest
> suggestion would be to make it a little more clear that MacPorts values its
> user community. There are some bug reports that go unresolved for a year or
> more, which makes it hard to stick with MacPorts if you're one of the users
> that keeps dealing with that bug. MacPorts also gives off the impression
> that it's hard to contribute (Homebrew has 3450 contributors according to
> GitHub, and is GitHub's most starred/forked repo; MacPorts has 182
> contributors according to Ohloh). There are a lot of similar complaints in
> "Homebrew vs MacPorts" posts, which is why Homebrew seems to be eating
> MacPorts' lunch, despite MacPorts' big head start, and initially far
> superior feature set.
> >
> > --
> > Geoffrey Oxberry, Ph.D., E.I.T.
> > goxberry at gmail.com
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/f4d65b53/attachment.html>

From balay at mcs.anl.gov  Tue Jan 28 12:29:31 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 12:29:31 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1401281222060.18968@asterix>

On Tue, 28 Jan 2014, Barry Smith wrote:

> > 
> > Satish is probably right here about the build location. It's been three or four years since I've installed it this way. I stand by that it's still difficult to revert. I actually tried this method because of PETSc and regretted it because the experience was terrible. Using a package manager is more maintainable, and I think PETSc's recommendation of the hpc.sourceforge build is a disservice to both users and to PETSc's excellent reputation.
> 
>    I think package managers for Mac OS are a disservice to the community and recommend not using them. (See all the discussions in these emails about how they fuck up).
> 

My view is: anyone using OSX has bought into the idea of not having a
proper package management system. [yeah you get easy-install packages
- but most of them don't have an proper way to uninstall - unless its
an "osx-app" which you can drag/drop into trash]

gfortran from hpc.sourceforge does things "no worse" than most packages
that are available for OSX.

Its not obvious - but one can use the file listing from the tarball [as
mentioned in my previous e-mail to uninstall]. And is tucked away
in /usr/local - so it doesn't do any damage like other packages.

[for eg: install mercurial for OSX - and see if you can uninstall it]

I agree a better package management system [aka macports/homebrew]
should be preferable. But with all the wierd issues that keep comping
up with users using macports on petsc lists - I can't convince myself
that it is a better recommendation.

perhaps homebrew is better - I don't know.

I would aswell recommend virtualbox with linux as a superior choice.

Satish


From jed at jedbrown.org  Tue Jan 28 12:51:37 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 12:51:37 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281222060.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
Message-ID: <87ppnclz2e.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> I agree a better package management system [aka macports/homebrew]
> should be preferable. But with all the wierd issues that keep comping
> up with users using macports on petsc lists - I can't convince myself
> that it is a better recommendation.

People either need a reliable way to upgrade or the stack will get out
of date.  Also, system upgrades will be stop-the-world events with
unknown system changes and the only safe thing is to reinstall the
entire stack.  Very few people can remember all the installation quirks
they have gone through, and even then, it takes time to reinstall
everything.

> I would aswell recommend virtualbox with linux as a superior choice.

I recommend

  # dd if=/dev/zero of=/dev/disk0

and install a decent operating system with a package manager on your
now-impeccably-clean disk.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/c82f4132/attachment.sig>

From knepley at gmail.com  Tue Jan 28 12:54:14 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 12:54:14 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87ppnclz2e.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
Message-ID: <CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>

On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> > I agree a better package management system [aka macports/homebrew]
> > should be preferable. But with all the wierd issues that keep comping
> > up with users using macports on petsc lists - I can't convince myself
> > that it is a better recommendation.
>
> People either need a reliable way to upgrade or the stack will get out
> of date.  Also, system upgrades will be stop-the-world events with
> unknown system changes and the only safe thing is to reinstall the
> entire stack.  Very few people can remember all the installation quirks
> they have gone through, and even then, it takes time to reinstall
> everything.


This is largely ideology. Some of us have used OSX for development for years
with little productivity impact.

  Matt


> > I would aswell recommend virtualbox with linux as a superior choice.
>
> I recommend
>
>   # dd if=/dev/zero of=/dev/disk0
>
> and install a decent operating system with a package manager on your
> now-impeccably-clean disk.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/9108c1dd/attachment.html>

From mfadams at lbl.gov  Tue Jan 28 13:10:24 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 28 Jan 2014 13:10:24 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
Message-ID: <CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>

I managed to get macports to rebuild.  It was not that hard as it turns
out, I was just confused about exactly what to do.  GDB broke but I was
able to install it after macports updated outdated packages.  PETSc still
seem to fail in MPICH, however.



On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Satish Balay <balay at mcs.anl.gov> writes:
>> > I agree a better package management system [aka macports/homebrew]
>> > should be preferable. But with all the wierd issues that keep comping
>> > up with users using macports on petsc lists - I can't convince myself
>> > that it is a better recommendation.
>>
>> People either need a reliable way to upgrade or the stack will get out
>> of date.  Also, system upgrades will be stop-the-world events with
>> unknown system changes and the only safe thing is to reinstall the
>> entire stack.  Very few people can remember all the installation quirks
>> they have gone through, and even then, it takes time to reinstall
>> everything.
>
>
> This is largely ideology. Some of us have used OSX for development for
> years
> with little productivity impact.
>
>   Matt
>
>
>>  > I would aswell recommend virtualbox with linux as a superior choice.
>>
>> I recommend
>>
>>   # dd if=/dev/zero of=/dev/disk0
>>
>> and install a decent operating system with a package manager on your
>> now-impeccably-clean disk.
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/7217f83c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 1322892 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/7217f83c/attachment.obj>

From balay at mcs.anl.gov  Tue Jan 28 13:26:47 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 13:26:47 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401281324440.18968@asterix>

I'll have to check back my e-mail archive - but I think
'/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?

I thought petsc configure prefred /usr/bin/make - if it was found.

One more -ve for macports..

Satish

------

================================================================================
TEST configureMake from config.packages.make(/Users/markadams/Codes/petsc/config/BuildSystem/config/packages/make.py:61)
TESTING: configureMake from config.packages.make(config/BuildSystem/config/packages/make.py:61)
  Check for user specified make - or gmake, make
Checking for program /opt/local/bin/gmake...found
          Defined make macro "MAKE" to "/opt/local/bin/gmake"
================================================================================
gmake[2]: *** read jobs pipe: No such file or directory.  Stop.
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: *** write jobserver: Broken pipe.  Stop.
gmake[1]: *** [all-recursive] Error 1
gmake: *** [all] Error 2
gmake: INTERNAL: Exiting with 1 jobserver tokens available; should be 7!




On Tue, 28 Jan 2014, Mark Adams wrote:

> I managed to get macports to rebuild.  It was not that hard as it turns
> out, I was just confused about exactly what to do.  GDB broke but I was
> able to install it after macports updated outdated packages.  PETSc still
> seem to fail in MPICH, however.
> 
> 
> 
> On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> > On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Satish Balay <balay at mcs.anl.gov> writes:
> >> > I agree a better package management system [aka macports/homebrew]
> >> > should be preferable. But with all the wierd issues that keep comping
> >> > up with users using macports on petsc lists - I can't convince myself
> >> > that it is a better recommendation.
> >>
> >> People either need a reliable way to upgrade or the stack will get out
> >> of date.  Also, system upgrades will be stop-the-world events with
> >> unknown system changes and the only safe thing is to reinstall the
> >> entire stack.  Very few people can remember all the installation quirks
> >> they have gone through, and even then, it takes time to reinstall
> >> everything.
> >
> >
> > This is largely ideology. Some of us have used OSX for development for
> > years
> > with little productivity impact.
> >
> >   Matt
> >
> >
> >>  > I would aswell recommend virtualbox with linux as a superior choice.
> >>
> >> I recommend
> >>
> >>   # dd if=/dev/zero of=/dev/disk0
> >>
> >> and install a decent operating system with a package manager on your
> >> now-impeccably-clean disk.
> >>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which their
> > experiments lead.
> > -- Norbert Wiener
> >
> 



From bsmith at mcs.anl.gov  Tue Jan 28 13:27:24 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 28 Jan 2014 13:27:24 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
Message-ID: <F2B4CC3B-BB35-4B4E-942C-0F5D1B494DB7@mcs.anl.gov>


On Jan 28, 2014, at 1:10 PM, Mark Adams <mfadams at lbl.gov> wrote:

> I managed to get macports to rebuild.  It was not that hard as it turns out, I was just confused about exactly what to do.  GDB broke but I was able to install it after macports updated outdated packages.  PETSc still seem to fail in MPICH, however.

   Correction, MPICH failed to install (probably for the same reason as before)

  Barry

> 
> 
> 
> On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> Satish Balay <balay at mcs.anl.gov> writes:
> > I agree a better package management system [aka macports/homebrew]
> > should be preferable. But with all the wierd issues that keep comping
> > up with users using macports on petsc lists - I can't convince myself
> > that it is a better recommendation.
> 
> People either need a reliable way to upgrade or the stack will get out
> of date.  Also, system upgrades will be stop-the-world events with
> unknown system changes and the only safe thing is to reinstall the
> entire stack.  Very few people can remember all the installation quirks
> they have gone through, and even then, it takes time to reinstall
> everything.
> 
> This is largely ideology. Some of us have used OSX for development for years
> with little productivity impact.
> 
>   Matt
>  
> > I would aswell recommend virtualbox with linux as a superior choice.
> 
> I recommend
> 
>   # dd if=/dev/zero of=/dev/disk0
> 
> and install a decent operating system with a package manager on your
> now-impeccably-clean disk.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> <configure.log>



From mfadams at lbl.gov  Tue Jan 28 13:53:52 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 28 Jan 2014 13:53:52 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281324440.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
Message-ID: <CADOhEh7LuTG0Lr8U2t6ezMJnz1MTHqvPRJn+B4NkmYrTbrdAxQ@mail.gmail.com>

On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> I'll have to check back my e-mail archive - but I think
> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
>
>
13:49  ~$ which make
/usr/bin/make
13:52  ~$ which gmake
/opt/local/bin/gmake
13:52  ~$

I don't recall what the fix was.  Should I uninstall gmake?


> I thought petsc configure prefred /usr/bin/make - if it was found.
>
> One more -ve for macports..
>
> Satish
>
> ------
>
>
> ================================================================================
> TEST configureMake from
> config.packages.make(/Users/markadams/Codes/petsc/config/BuildSystem/config/packages/make.py:61)
> TESTING: configureMake from
> config.packages.make(config/BuildSystem/config/packages/make.py:61)
>   Check for user specified make - or gmake, make
> Checking for program /opt/local/bin/gmake...found
>           Defined make macro "MAKE" to "/opt/local/bin/gmake"
>
> ================================================================================
> gmake[2]: *** read jobs pipe: No such file or directory.  Stop.
> gmake[2]: *** Waiting for unfinished jobs....
> gmake[2]: *** write jobserver: Broken pipe.  Stop.
> gmake[1]: *** [all-recursive] Error 1
> gmake: *** [all] Error 2
> gmake: INTERNAL: Exiting with 1 jobserver tokens available; should be 7!
>
>
>
>
> On Tue, 28 Jan 2014, Mark Adams wrote:
>
> > I managed to get macports to rebuild.  It was not that hard as it turns
> > out, I was just confused about exactly what to do.  GDB broke but I was
> > able to install it after macports updated outdated packages.  PETSc still
> > seem to fail in MPICH, however.
> >
> >
> >
> > On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> > > On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> > >
> > >> Satish Balay <balay at mcs.anl.gov> writes:
> > >> > I agree a better package management system [aka macports/homebrew]
> > >> > should be preferable. But with all the wierd issues that keep
> comping
> > >> > up with users using macports on petsc lists - I can't convince
> myself
> > >> > that it is a better recommendation.
> > >>
> > >> People either need a reliable way to upgrade or the stack will get out
> > >> of date.  Also, system upgrades will be stop-the-world events with
> > >> unknown system changes and the only safe thing is to reinstall the
> > >> entire stack.  Very few people can remember all the installation
> quirks
> > >> they have gone through, and even then, it takes time to reinstall
> > >> everything.
> > >
> > >
> > > This is largely ideology. Some of us have used OSX for development for
> > > years
> > > with little productivity impact.
> > >
> > >   Matt
> > >
> > >
> > >>  > I would aswell recommend virtualbox with linux as a superior
> choice.
> > >>
> > >> I recommend
> > >>
> > >>   # dd if=/dev/zero of=/dev/disk0
> > >>
> > >> and install a decent operating system with a package manager on your
> > >> now-impeccably-clean disk.
> > >>
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/043ccc73/attachment.html>

From mfadams at lbl.gov  Tue Jan 28 13:58:18 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 28 Jan 2014 13:58:18 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281324440.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
Message-ID: <CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>

Oh yes.  I had a use-make=make commended out. I uncommented it and its
seems to have gotten MPICH installed.


On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> I'll have to check back my e-mail archive - but I think
> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
>
> I thought petsc configure prefred /usr/bin/make - if it was found.
>
> One more -ve for macports..
>
> Satish
>
> ------
>
>
> ================================================================================
> TEST configureMake from
> config.packages.make(/Users/markadams/Codes/petsc/config/BuildSystem/config/packages/make.py:61)
> TESTING: configureMake from
> config.packages.make(config/BuildSystem/config/packages/make.py:61)
>   Check for user specified make - or gmake, make
> Checking for program /opt/local/bin/gmake...found
>           Defined make macro "MAKE" to "/opt/local/bin/gmake"
>
> ================================================================================
> gmake[2]: *** read jobs pipe: No such file or directory.  Stop.
> gmake[2]: *** Waiting for unfinished jobs....
> gmake[2]: *** write jobserver: Broken pipe.  Stop.
> gmake[1]: *** [all-recursive] Error 1
> gmake: *** [all] Error 2
> gmake: INTERNAL: Exiting with 1 jobserver tokens available; should be 7!
>
>
>
>
> On Tue, 28 Jan 2014, Mark Adams wrote:
>
> > I managed to get macports to rebuild.  It was not that hard as it turns
> > out, I was just confused about exactly what to do.  GDB broke but I was
> > able to install it after macports updated outdated packages.  PETSc still
> > seem to fail in MPICH, however.
> >
> >
> >
> > On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> > > On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> > >
> > >> Satish Balay <balay at mcs.anl.gov> writes:
> > >> > I agree a better package management system [aka macports/homebrew]
> > >> > should be preferable. But with all the wierd issues that keep
> comping
> > >> > up with users using macports on petsc lists - I can't convince
> myself
> > >> > that it is a better recommendation.
> > >>
> > >> People either need a reliable way to upgrade or the stack will get out
> > >> of date.  Also, system upgrades will be stop-the-world events with
> > >> unknown system changes and the only safe thing is to reinstall the
> > >> entire stack.  Very few people can remember all the installation
> quirks
> > >> they have gone through, and even then, it takes time to reinstall
> > >> everything.
> > >
> > >
> > > This is largely ideology. Some of us have used OSX for development for
> > > years
> > > with little productivity impact.
> > >
> > >   Matt
> > >
> > >
> > >>  > I would aswell recommend virtualbox with linux as a superior
> choice.
> > >>
> > >> I recommend
> > >>
> > >>   # dd if=/dev/zero of=/dev/disk0
> > >>
> > >> and install a decent operating system with a package manager on your
> > >> now-impeccably-clean disk.
> > >>
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/dcbb65b2/attachment.html>

From balay at mcs.anl.gov  Tue Jan 28 14:01:41 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 14:01:41 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401281400000.18968@asterix>

Yeah --with-make=/usr/bin/make is the workarround.

I'll have to recheck why gmake is prefered over /usr/bin/make. [I
thought I fixed that].

Also can you check if --download-make works or breaks similarly?

Satish

On Tue, 28 Jan 2014, Mark Adams wrote:

> Oh yes.  I had a use-make=make commended out. I uncommented it and its
> seems to have gotten MPICH installed.
> 
> 
> On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > I'll have to check back my e-mail archive - but I think
> > '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
> >
> > I thought petsc configure prefred /usr/bin/make - if it was found.
> >
> > One more -ve for macports..
> >
> > Satish
> >
> > ------
> >
> >
> > ================================================================================
> > TEST configureMake from
> > config.packages.make(/Users/markadams/Codes/petsc/config/BuildSystem/config/packages/make.py:61)
> > TESTING: configureMake from
> > config.packages.make(config/BuildSystem/config/packages/make.py:61)
> >   Check for user specified make - or gmake, make
> > Checking for program /opt/local/bin/gmake...found
> >           Defined make macro "MAKE" to "/opt/local/bin/gmake"
> >
> > ================================================================================
> > gmake[2]: *** read jobs pipe: No such file or directory.  Stop.
> > gmake[2]: *** Waiting for unfinished jobs....
> > gmake[2]: *** write jobserver: Broken pipe.  Stop.
> > gmake[1]: *** [all-recursive] Error 1
> > gmake: *** [all] Error 2
> > gmake: INTERNAL: Exiting with 1 jobserver tokens available; should be 7!
> >
> >
> >
> >
> > On Tue, 28 Jan 2014, Mark Adams wrote:
> >
> > > I managed to get macports to rebuild.  It was not that hard as it turns
> > > out, I was just confused about exactly what to do.  GDB broke but I was
> > > able to install it after macports updated outdated packages.  PETSc still
> > > seem to fail in MPICH, however.
> > >
> > >
> > >
> > > On Tue, Jan 28, 2014 at 12:54 PM, Matthew Knepley <knepley at gmail.com>
> > wrote:
> > >
> > > > On Tue, Jan 28, 2014 at 12:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> > > >
> > > >> Satish Balay <balay at mcs.anl.gov> writes:
> > > >> > I agree a better package management system [aka macports/homebrew]
> > > >> > should be preferable. But with all the wierd issues that keep
> > comping
> > > >> > up with users using macports on petsc lists - I can't convince
> > myself
> > > >> > that it is a better recommendation.
> > > >>
> > > >> People either need a reliable way to upgrade or the stack will get out
> > > >> of date.  Also, system upgrades will be stop-the-world events with
> > > >> unknown system changes and the only safe thing is to reinstall the
> > > >> entire stack.  Very few people can remember all the installation
> > quirks
> > > >> they have gone through, and even then, it takes time to reinstall
> > > >> everything.
> > > >
> > > >
> > > > This is largely ideology. Some of us have used OSX for development for
> > > > years
> > > > with little productivity impact.
> > > >
> > > >   Matt
> > > >
> > > >
> > > >>  > I would aswell recommend virtualbox with linux as a superior
> > choice.
> > > >>
> > > >> I recommend
> > > >>
> > > >>   # dd if=/dev/zero of=/dev/disk0
> > > >>
> > > >> and install a decent operating system with a package manager on your
> > > >> now-impeccably-clean disk.
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > What most experimenters take for granted before they begin their
> > > > experiments is infinitely more interesting than any results to which
> > their
> > > > experiments lead.
> > > > -- Norbert Wiener
> > > >
> > >
> >
> >
> 



From jed at jedbrown.org  Tue Jan 28 14:03:55 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 14:03:55 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281400000.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<alpine.LFD.2.11.14012 81400000 .18968@asterix>
Message-ID: <87fvo7naac.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> Yeah --with-make=/usr/bin/make is the workarround.
>
> I'll have to recheck why gmake is prefered over /usr/bin/make. [I
> thought I fixed that].

What is it doing wrong?  A broken gmake needs to be reported to whoever
is responsible (macports?).

Preferring gmake when present is safer for BSD systems where "make" is
usually BSD make.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/4fe48374/attachment.sig>

From sean.michael.farley at gmail.com  Tue Jan 28 14:09:03 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 14:09:03 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2. 11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
Message-ID: <m2fvo73m3k.fsf@gmail.com>


mfadams at lbl.gov writes:

> Oh yes.  I had a use-make=make commended out. I uncommented it and its
> seems to have gotten MPICH installed.

There is a newly (only a few days old) updated mpich port that I now use
with my petsc install.

> On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> I'll have to check back my e-mail archive - but I think
>> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
>>
>> I thought petsc configure prefred /usr/bin/make - if it was found.
>>
>> One more -ve for macports..

There is a fundamental problem with using two stacks. Similar, I think,
to trying to simultaneously link to libc++ and libstdc++. It is just
asking for trouble.


From balay at mcs.anl.gov  Tue Jan 28 14:10:07 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 14:10:07 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87fvo7naac.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<alpine.LFD.2.11.14012 81400000 .18968@asterix>
	<87fvo7naac.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401281405470.18968@asterix>

On Tue, 28 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > Yeah --with-make=/usr/bin/make is the workarround.
> >
> > I'll have to recheck why gmake is prefered over /usr/bin/make. [I
> > thought I fixed that].
> 
> What is it doing wrong?  A broken gmake needs to be reported to whoever
> is responsible (macports?).

yes.
> 
> Preferring gmake when present is safer for BSD systems where "make" is
> usually BSD make.

Yes - I meant to do the following [which shold be eqivalent on bsd -and workarround this issue]

if isGnumake('make') : self.make = 'make'
else if isGnumake('gmake'): self.make = 'gmake'
else error('require gnumake')

Satish


From balay at mcs.anl.gov  Tue Jan 28 14:13:57 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 14:13:57 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281405470.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<alpine.LFD.2.11.14012 81400000 .18968@asterix>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
Message-ID: <alpine.LFD.2.11.1401281410300.18968@asterix>

On Tue, 28 Jan 2014, Satish Balay wrote:

> On Tue, 28 Jan 2014, Jed Brown wrote:
> 
> > Satish Balay <balay at mcs.anl.gov> writes:
> > 
> > > Yeah --with-make=/usr/bin/make is the workarround.
> > >
> > > I'll have to recheck why gmake is prefered over /usr/bin/make. [I
> > > thought I fixed that].
> > 
> > What is it doing wrong?  A broken gmake needs to be reported to whoever
> > is responsible (macports?).
> 
> yes.

BTW: previously this issue came up for Mark and Barry. But Sean couldn't reproduce.

http:/http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-September/013167.html

Satish


From aron at ahmadia.net  Tue Jan 28 14:16:28 2014
From: aron at ahmadia.net (Aron Ahmadia)
Date: Tue, 28 Jan 2014 15:16:28 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281410300.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
Message-ID: <CAPhiW4gx-ckvAZfDjCAqGDtPaQikj6nX83yaGBrmuAo1i5183Q@mail.gmail.com>

FWIW, on OS X.9, the venerable make 3.81 is installed natively and does not
seem to cause much grief, despite its age.


On Tue, Jan 28, 2014 at 3:13 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 28 Jan 2014, Satish Balay wrote:
>
> > On Tue, 28 Jan 2014, Jed Brown wrote:
> >
> > > Satish Balay <balay at mcs.anl.gov> writes:
> > >
> > > > Yeah --with-make=/usr/bin/make is the workarround.
> > > >
> > > > I'll have to recheck why gmake is prefered over /usr/bin/make. [I
> > > > thought I fixed that].
> > >
> > > What is it doing wrong?  A broken gmake needs to be reported to whoever
> > > is responsible (macports?).
> >
> > yes.
>
> BTW: previously this issue came up for Mark and Barry. But Sean couldn't
> reproduce.
>
> http:/
> http://lists.mcs.anl.gov/pipermail/petsc-dev/2013-September/013167.html
>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/11a6524f/attachment.html>

From mfadams at lbl.gov  Tue Jan 28 14:24:58 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 28 Jan 2014 14:24:58 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281400000.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<alpine.LFD.2.11.1401281400000.18968@asterix>
Message-ID: <CADOhEh4=iDdEtkQFYiQBqKHcmVMuS_VbLkgPj3yhFGohJX4YzA@mail.gmail.com>

>
>
> Also can you check if --download-make works or breaks similarly?
>
>

This works.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/3865160e/attachment.html>

From balay at mcs.anl.gov  Tue Jan 28 14:39:03 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 14:39:03 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh4=iDdEtkQFYiQBqKHcmVMuS_VbLkgPj3yhFGohJX4YzA@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<alpine.LFD.2.11.1401281400000.18968@asterix>
	<CADOhEh4=iDdEtkQFYiQBqKHcmVMuS_VbLkgPj3yhFGohJX4YzA@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401281438410.18968@asterix>

On Tue, 28 Jan 2014, Mark Adams wrote:

> >
> >
> > Also can you check if --download-make works or breaks similarly?

> This works.

Thanks. Then the problem is primarily with the macports build of gmake

Satish



From balay at mcs.anl.gov  Tue Jan 28 14:40:09 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 14:40:09 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2fvo73m3k.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2. 11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
Message-ID: <alpine.LFD.2.11.1401281417480.18968@asterix>

On Tue, 28 Jan 2014, Sean Farley wrote:

> 
> mfadams at lbl.gov writes:
> 
> > Oh yes.  I had a use-make=make commended out. I uncommented it and its
> > seems to have gotten MPICH installed.
> 
> There is a newly (only a few days old) updated mpich port that I now use
> with my petsc install.

This is not really a fix - as the 'gmake' error will pop up with petsc
part of the build aswell.

> 
> > On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> >> I'll have to check back my e-mail archive - but I think
> >> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
> >>
> >> I thought petsc configure prefred /usr/bin/make - if it was found.
> >>
> >> One more -ve for macports..
> 
> There is a fundamental problem with using two stacks. Similar, I think,
> to trying to simultaneously link to libc++ and libstdc++. It is just
> asking for trouble.

I'm not sure how this is causing gmake 'internal' error..

Satish


From rupp at mcs.anl.gov  Tue Jan 28 14:58:24 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 28 Jan 2014 21:58:24 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <CAJzU6sAQ+jSebyp5G1uk9+oqMDMCHHSX7x9pU7fuWupEf9-Quw@mail.gmail.com>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>	<52E773D6.8040407@mcs.anl.gov>	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>	<52E7D70B.3050107@mcs.anl.gov>	<CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>
	<CAJzU6sAQ+jSebyp5G1uk9+oqMDMCHHSX7x9pU7fuWupEf9-Quw@mail.gmail.com>
Message-ID: <52E819F0.9000400@mcs.anl.gov>

Hi Mani,

 > Note that the error occurs in the middle of the run, after a certain
> number of time steps. The time step at which it fails is different for
> the two machines for which I tested. And it only happens when I use the
> GPU. The OpenCL backend using a CPU with Intel opencl works fine.

what exactly do you mean by 'works fine' for the Intel OpenCL SDK? Do 
you see an increase in memory, or is the memory consumption constant 
throught the run? You won't necessarily see an allocation failure 
because your machine will most likely have enough main RAM to compensate.

Best regards,
Karli



From mc0710 at gmail.com  Tue Jan 28 15:07:49 2014
From: mc0710 at gmail.com (Mani Chandra)
Date: Tue, 28 Jan 2014 15:07:49 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <52E819F0.9000400@mcs.anl.gov>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>
	<52E773D6.8040407@mcs.anl.gov>
	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>
	<52E7D70B.3050107@mcs.anl.gov>
	<CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>
	<CAJzU6sAQ+jSebyp5G1uk9+oqMDMCHHSX7x9pU7fuWupEf9-Quw@mail.gmail.com>
	<52E819F0.9000400@mcs.anl.gov>
Message-ID: <CAJzU6sBVtKsNnhjgCfBmkvOQudX+gZTB7gR8-cHrvdDD0jOvCA@mail.gmail.com>

Sorry, you're right. Indeed even when I use the Intel OpenCL to run on the
CPU, the memory usage keeps increasing as the run progresses but the CPU
RAM is so much that the run doesn't crash till the time I ran it. So
basically there is a memory leak.


On Tue, Jan 28, 2014 at 2:58 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
> > Note that the error occurs in the middle of the run, after a certain
>
>> number of time steps. The time step at which it fails is different for
>> the two machines for which I tested. And it only happens when I use the
>> GPU. The OpenCL backend using a CPU with Intel opencl works fine.
>>
>
> what exactly do you mean by 'works fine' for the Intel OpenCL SDK? Do you
> see an increase in memory, or is the memory consumption constant throught
> the run? You won't necessarily see an allocation failure because your
> machine will most likely have enough main RAM to compensate.
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/cd445a49/attachment.html>

From rupp at mcs.anl.gov  Tue Jan 28 15:22:13 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Tue, 28 Jan 2014 22:22:13 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
	(continued)
In-Reply-To: <CAJzU6sBVtKsNnhjgCfBmkvOQudX+gZTB7gR8-cHrvdDD0jOvCA@mail.gmail.com>
References: <CAJzU6sBC_q0zYaZ5uGPEh7BQkDcY-=_n8QnTvhCen6HZPtj4XQ@mail.gmail.com>	<52E773D6.8040407@mcs.anl.gov>	<CAJzU6sBikr+tv8-QKATMRxSdzZgT_S4ve9JJgYwGph3mO7HnLA@mail.gmail.com>	<52E7D70B.3050107@mcs.anl.gov>	<CAJzU6sDb2UcKFr8h=u_6m=rpE2jZB8oSWoh2bMc=doX6+VfrYA@mail.gmail.com>	<CAJzU6sAQ+jSebyp5G1uk9+oqMDMCHHSX7x9pU7fuWupEf9-Quw@mail.gmail.com>	<52E819F0.9000400@mcs.anl.gov>
	<CAJzU6sBVtKsNnhjgCfBmkvOQudX+gZTB7gR8-cHrvdDD0jOvCA@mail.gmail.com>
Message-ID: <52E81F85.3090703@mcs.anl.gov>

Hi Mani,

 > Sorry, you're right. Indeed even when I use the Intel OpenCL to run on
> the CPU, the memory usage keeps increasing as the run progresses but the
> CPU RAM is so much that the run doesn't crash till the time I ran it. So
> basically there is a memory leak.

Thanks, that certainly helps with the diagnostics. :-)

Best regards,
Karli




> On Tue, Jan 28, 2014 at 2:58 PM, Karl Rupp <rupp at mcs.anl.gov
> <mailto:rupp at mcs.anl.gov>> wrote:
>
>     Hi Mani,
>
>
>      > Note that the error occurs in the middle of the run, after a certain
>
>         number of time steps. The time step at which it fails is
>         different for
>         the two machines for which I tested. And it only happens when I
>         use the
>         GPU. The OpenCL backend using a CPU with Intel opencl works fine.
>
>
>     what exactly do you mean by 'works fine' for the Intel OpenCL SDK?
>     Do you see an increase in memory, or is the memory consumption
>     constant throught the run? You won't necessarily see an allocation
>     failure because your machine will most likely have enough main RAM
>     to compensate.
>
>     Best regards,
>     Karli
>
>



From irving at naml.us  Tue Jan 28 15:31:08 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 13:31:08 -0800
Subject: [petsc-dev] loop structure in PetscFEIntegrateJacobian_Basic
Message-ID: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>

PetscFEIntegrateJacobian_Basic has four layers of loops:

   e - element
   f - component of field being differentiated
   g - component of field we're differentiating against
   q - quadrature point

It looks like g3_func and friends are being called once for each
combination of (e,f,g,q), even though the calls to g3_func are
independent of (f,g).  If g3_func is expensive, this duplication seems
a bit slow.

It seems like the loops should be flipped to be (e,q,f,g) major order,
with the user functions hoisted up to the (e,q) level.  Is there a
reason not to do this?

Geoffrey


From irving at naml.us  Tue Jan 28 15:33:28 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 13:33:28 -0800
Subject: [petsc-dev] loop structure in PetscFEIntegrateJacobian_Basic
In-Reply-To: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>
References: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>
Message-ID: <CAJ1ofpear5=9wd4V-Xau3m66uHftqi6+GhGZHHnyCy1h5Gj=qg@mail.gmail.com>

On Tue, Jan 28, 2014 at 1:31 PM, Geoffrey Irving <irving at naml.us> wrote:
> PetscFEIntegrateJacobian_Basic has four layers of loops:
>
>    e - element
>    f - component of field being differentiated
>    g - component of field we're differentiating against
>    q - quadrature point

Typo: f and g index basis functions, not field components.

> It looks like g3_func and friends are being called once for each
> combination of (e,f,g,q), even though the calls to g3_func are
> independent of (f,g).  If g3_func is expensive, this duplication seems
> a bit slow.
>
> It seems like the loops should be flipped to be (e,q,f,g) major order,
> with the user functions hoisted up to the (e,q) level.  Is there a
> reason not to do this?
>
> Geoffrey


From knepley at gmail.com  Tue Jan 28 15:45:40 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 15:45:40 -0600
Subject: [petsc-dev] loop structure in PetscFEIntegrateJacobian_Basic
In-Reply-To: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>
References: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>
Message-ID: <CAMYG4G=BKadvA4g_S+DLuExbkocE7=aqUf81QnL1zsy7OomQRA@mail.gmail.com>

On Tue, Jan 28, 2014 at 3:31 PM, Geoffrey Irving <irving at naml.us> wrote:

> PetscFEIntegrateJacobian_Basic has four layers of loops:
>
>    e - element
>    f - component of field being differentiated
>    g - component of field we're differentiating against
>    q - quadrature point
>
> It looks like g3_func and friends are being called once for each
> combination of (e,f,g,q), even though the calls to g3_func are
> independent of (f,g).  If g3_func is expensive, this duplication seems
> a bit slow.
>
> It seems like the loops should be flipped to be (e,q,f,g) major order,
> with the user functions hoisted up to the (e,q) level.  Is there a
> reason not to do this?


Yes, it is stupid. I will fix it.

   Matt


>
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/084583b8/attachment.html>

From irving at naml.us  Tue Jan 28 15:55:45 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 13:55:45 -0800
Subject: [petsc-dev] attempt at understanding PetscFEM
In-Reply-To: <87txf9teas.fsf@jedbrown.org>
References: <CAJ1ofpdD7CVGiaApA7_0tuNru=KL0PN_DVw0b1OqO_5YYR7qcA@mail.gmail.com>
	<87zjp1tfql.fsf@jedbrown.org>
	<CAJ1ofpdY3JjtqjOemXSeAtJL2-gpMnLrDh_9+zTM9s6_XisgBA@mail.gmail.com>
	<87txf9teas.fsf@jedbrown.org>
Message-ID: <CAJ1ofpetTuRNORdVsbhmYpTgjZK98ArueAbQ+oOQwWRrt5EcYg@mail.gmail.com>

On Mon, Nov 18, 2013 at 4:39 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>
>> The seems to be a typo in the abstract:
>>
>>     This effect typically limits order to at most *quadratic*, despite
>> the favorable accuracy and stability properties offered by *quadratic*
>> and higher order discretization
>
> I guess it sounds funny, but quadratic is the crossover point: too
> expensive according to many people, but required by others.
>
>> The index notation was the whole point, incidentally, since it
>> transparently defines the order of array dimensions in the code.
>
> Okay, that is a convention that I think you got right, but there are
> vectorization reasons to change it in some settings, so if you're going
> to build something significant, it would be worth hedging against index
> ordering changes.

For what it's worth, I had actually gotten the convention wrong.
According to my document, g3 in the FEM case looked like

    g3(a,b,c,d) = d(P(a,b))/d(F(c,d))

but it's actually

    g3(a,b,c,d) = d(P(a,c))/d(F(b,d))

Geoffrey


From irving at naml.us  Tue Jan 28 17:05:19 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 15:05:19 -0800
Subject: [petsc-dev] resurrecting the finite element energy functions
	thread
In-Reply-To: <87r481gqid.fsf@jedbrown.org>
References: <CAJ1ofpfBagEqqM638PFPY8tZg14BL3zg1NdPQpQeomT9nqNngg@mail.gmail.com>
	<8738kju0hz.fsf@jedbrown.org>
	<CAJ1ofpeAiVprdVpjL_9R8-NpOMqv7RR8cdNP5+h05AmPS9N=zw@mail.gmail.com>
	<8738kjqy3c.fsf@jedbrown.org>
	<CAJ1ofpfN=_jAfgiOn0gGxFGJs4wsSd=qu6r59YpekrDenAN5FA@mail.gmail.com>
	<87mwirpiq3.fsf@jedbrown.org>
	<CAJ1ofpd-E02zyFd2rVgeOf9KaTTiRS3agaPhqqbZGM5p1xZG0Q@mail.gmail.com>
	<87r481gqid.fsf@jedbrown.org>
Message-ID: <CAJ1ofpcVaYeSCFUDy4mMVzsGJ0L546fx+RPfozqEcVwQ6EUHjQ@mail.gmail.com>

On Tue, Jan 21, 2014 at 10:04 AM, Jed Brown <jed at jedbrown.org> wrote:
> Geoffrey Irving <irving at naml.us> writes:
>
>> On Sun, Jan 19, 2014 at 5:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> Geoffrey Irving <irving at naml.us> writes:
>>>> It isn't, the only difference is that the function I'm imagining needs
>>>> an extra quadrature rule argument.  Nothing here is complicated, so if
>>>> there are no particular opinions I'll write the function and send a
>>>> patch.  Can I write it hard coded to MPI_SUM first, and we can discuss
>>>> how to generalize it after that?  I am less clear on how to best write
>>>> the generalized version.
>>>
>>> That's fine with me.
>>
>> Should the result be an array of PetscScalar or PetscReal?  My
>> application is for objectives, so PetscReal, but PetscScalar is more
>> generic.
>
> I would use PetscScalar.

I uploaded the branch:

    irving/fe-scalars

There are no unit tests in petsc yet, but I do have unit tests outside
of petsc which pass.

Geoffrey


From sean.michael.farley at gmail.com  Tue Jan 28 17:18:37 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 17:18:37 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
Message-ID: <m2d2jb3dbm.fsf@gmail.com>


bsmith at mcs.anl.gov writes:

> On Jan 28, 2014, at 12:14 PM, Geoff Oxberry <goxberry at gmail.com> wrote:
>
>> 
>> 
>> 
>> On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
>> 
>> goxberry at gmail.com writes:
>> 
>> > To echo what Aron said, I wouldn't point people at the
>> > hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
>> > it's a pain in the ass
>> 
>> Satish is probably right here about the build location. It's been three or four years since I've installed it this way. I stand by that it's still difficult to revert. I actually tried this method because of PETSc and regretted it because the experience was terrible. Using a package manager is more maintainable, and I think PETSc's recommendation of the hpc.sourceforge build is a disservice to both users and to PETSc's excellent reputation.
>
>    I think package managers for Mac OS are a disservice to the community and recommend not using them. (See all the discussions in these emails about how they fuck up).

Sigh. It is this type of curmudgeon behavior that pushes away people
from helping out with these type of projects. Packagers are just
volunteers and to estrange the current three (yes, three) would be
unfortunate. Not many (read: none) of the other devs care about having
multiple compilers (thanks, fortran) nor pandering to the scientific
community's lack of good software practices.

It is no secret that MacPorts has historically flubbed on lots of
PETSc-related issues. I have been trying to change this perspective
but this email thread pretty succinctly explains what makes my job
difficult.

Just look at how difficult it is to install these packages: superlu,
superlu_dist, metis, parmetis, scotch, scalapack, and mumps.

The comments here do nothing but drive away users and frustrate
potential collaborators. Not just for PETSc but for any project that
depends on PETSc (SLEPc, FEniCS, MOOSE, etc). The true disservice to the
community is forcing each user to manage their own packages.

Instead of criticizing here, the energy could be better spent by
contributing.


From goxberry at gmail.com  Tue Jan 28 17:31:43 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Tue, 28 Jan 2014 15:31:43 -0800
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281222060.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
Message-ID: <CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>

On Tuesday, January 28, 2014, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 28 Jan 2014, Barry Smith wrote:
>
> > >
> > > Satish is probably right here about the build location. It's been
> three or four years since I've installed it this way. I stand by that it's
> still difficult to revert. I actually tried this method because of PETSc
> and regretted it because the experience was terrible. Using a package
> manager is more maintainable, and I think PETSc's recommendation of the
> hpc.sourceforge build is a disservice to both users and to PETSc's
> excellent reputation.
> >
> >    I think package managers for Mac OS are a disservice to the community
> and recommend not using them. (See all the discussions in these emails
> about how they fuck up).
> >
>
> My view is: anyone using OSX has bought into the idea of not having a
> proper package management system. [yeah you get easy-install packages
> - but most of them don't have an proper way to uninstall - unless its
> an "osx-app" which you can drag/drop into trash]
>
> gfortran from hpc.sourceforge does things "no worse" than most packages
> that are available for OSX.
>
> Its not obvious - but one can use the file listing from the tarball [as
> mentioned in my previous e-mail to uninstall]. And is tucked away
> in /usr/local - so it doesn't do any damage like other packages.


If you follow the relative install paths, yes, your method does the job.
It's what I did, with some testing, and then I reinstalled the XCode
Command Line Tools. As long as a user is careful, no harm done; it's also
easy to mess up. You could do something similar via lsbom for packages in
OS X and/or delete via the package identifier using pkgutil and largely
only leave behind plist files. This method is what homebrew-cask uses for
managing OS X binaries.


> [for eg: install mercurial for OSX - and see if you can uninstall it]


For Mercurial, I'd install & uninstall via pip and a virtualenv. Your point
is well-taken for packages in general, and can still be managed (see
above). Uninstall apps will also do the job.


> I agree a better package management system [aka macports/homebrew]
> should be preferable. But with all the wierd issues that keep comping
> up with users using macports on petsc lists - I can't convince myself
> that it is a better recommendation.
>
> perhaps homebrew is better - I don't know.


You guys do the support work (and do a good job of it), so I defer to your
judgment here. My opinion was borne out of a bad experience and trying
several methods of installation; I apologize for the brief burst of
impertinence.

I would not recommend MacPorts either; I don't use it. Sean knows more
about this than I do and can better defend MacPorts.

With Homebrew, it should be possible to replicate the current
recommendation and only install six additional packages (cloog, gmp, isl,
mpfr, libmpc, and pkg-config).

Mixing gfortran 4.8 with the rest of the gcc 4.2 or clang 3.3 stack can
also be tricky, which was my point about different compiler versions. SciPy
recommends the AT&T build and states that problems may arise with the
hpc.sourceforge build; as you pointed out, this version causes problems for
PETSc. So I tend to use multiple builds of PETSc in different PETSC_ARCH
directories; one of these builds is a gcc 4.8 build because PETSc is
relatively self-contained (which is a testament to your design and build
system), so I'm not terribly worried about system library conflicts.

My Python rant is not a good argument for hpc.sourceforge (or against
package managers) because sitewide installs of interpreted language
packages -- especially via an OS package manager -- should be minimized,
regardless of operating system or distribution. Otherwise, you run into
problems like pip trying to overwrite system libraries.

I would aswell recommend virtualbox with linux as a superior choice.
>
> Satish
>

Probably the best choice. Even then, some can't run virtual machines (I
can't at work), and OS X is a lesser evil than a Windows machine.

As Jed points out, manual installation is difficult to maintain. Fixing
a package manager is my best path forward (not getting a different job);
they're not perfect, but they won't get better unless people work on
them. I agree with Matt that most of this discussion is ideology.


-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/acc924ba/attachment.html>

From bsmith at mcs.anl.gov  Tue Jan 28 17:32:37 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 28 Jan 2014 17:32:37 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2d2jb3dbm.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<m2d2jb3dbm.fsf@gmail.com>
Message-ID: <363B32DF-32C2-4C14-AACE-2640EE66BAE5@mcs.anl.gov>


  I think that Jed could reasonably argue that the basic Mac OS infrastructure is so screwy that trying to maintain a reasonable open source package manager on top of it is a lost cause; though I love Mac OS dearly, I would have to agree with him.

  You guys are trying to build a castle in a swamp (which itself has shifting sands), until Apple makes something upon which a reasonable open source package manager could be built you will always have basic troubles. Apple doesn?t seem to know or care 

  Barry



On Jan 28, 2014, at 5:18 PM, Sean Farley <sean.michael.farley at gmail.com> wrote:

> 
> bsmith at mcs.anl.gov writes:
> 
>> On Jan 28, 2014, at 12:14 PM, Geoff Oxberry <goxberry at gmail.com> wrote:
>> 
>>> 
>>> 
>>> 
>>> On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley <sean.michael.farley at gmail.com> wrote:
>>> 
>>> goxberry at gmail.com writes:
>>> 
>>>> To echo what Aron said, I wouldn't point people at the
>>>> hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
>>>> it's a pain in the ass
>>> 
>>> Satish is probably right here about the build location. It's been three or four years since I've installed it this way. I stand by that it's still difficult to revert. I actually tried this method because of PETSc and regretted it because the experience was terrible. Using a package manager is more maintainable, and I think PETSc's recommendation of the hpc.sourceforge build is a disservice to both users and to PETSc's excellent reputation.
>> 
>>   I think package managers for Mac OS are a disservice to the community and recommend not using them. (See all the discussions in these emails about how they fuck up).
> 
> Sigh. It is this type of curmudgeon behavior that pushes away people
> from helping out with these type of projects. Packagers are just
> volunteers and to estrange the current three (yes, three) would be
> unfortunate. Not many (read: none) of the other devs care about having
> multiple compilers (thanks, fortran) nor pandering to the scientific
> community's lack of good software practices.
> 
> It is no secret that MacPorts has historically flubbed on lots of
> PETSc-related issues. I have been trying to change this perspective
> but this email thread pretty succinctly explains what makes my job
> difficult.
> 
> Just look at how difficult it is to install these packages: superlu,
> superlu_dist, metis, parmetis, scotch, scalapack, and mumps.
> 
> The comments here do nothing but drive away users and frustrate
> potential collaborators. Not just for PETSc but for any project that
> depends on PETSc (SLEPc, FEniCS, MOOSE, etc). The true disservice to the
> community is forcing each user to manage their own packages.
> 
> Instead of criticizing here, the energy could be better spent by
> contributing.



From sean.michael.farley at gmail.com  Tue Jan 28 17:34:54 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 17:34:54 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281417480.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
Message-ID: <m2a9ef3ckh.fsf@gmail.com>


balay at mcs.anl.gov writes:

> On Tue, 28 Jan 2014, Sean Farley wrote:
>
>> 
>> mfadams at lbl.gov writes:
>> 
>> > Oh yes.  I had a use-make=make commended out. I uncommented it and its
>> > seems to have gotten MPICH installed.
>> 
>> There is a newly (only a few days old) updated mpich port that I now use
>> with my petsc install.
>
> This is not really a fix - as the 'gmake' error will pop up with petsc
> part of the build aswell.

False. It works perfectly fine here.
>> 
>> > On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>> >
>> >> I'll have to check back my e-mail archive - but I think
>> >> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
>> >>
>> >> I thought petsc configure prefred /usr/bin/make - if it was found.
>> >>
>> >> One more -ve for macports..
>> 
>> There is a fundamental problem with using two stacks. Similar, I think,
>> to trying to simultaneously link to libc++ and libstdc++. It is just
>> asking for trouble.
>
> I'm not sure how this is causing gmake 'internal' error..

It doesn't necessarily mean that gmake is broken but could be something
mismatched. I can't tell without the internal logs.

Though, I recommend that Mark not use any --download options through
petsc because those packages are now in MacPorts and mixing two stacks
is just asking for trouble.


From goxberry at gmail.com  Tue Jan 28 17:38:21 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Tue, 28 Jan 2014 15:38:21 -0800
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2d2jb3dbm.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<m2d2jb3dbm.fsf@gmail.com>
Message-ID: <CAHC8rGtkf_PLx+0x7AgQcGMJ5g4CsRi=MwZjOJwAVM=vRUMPbg@mail.gmail.com>

On Tuesday, January 28, 2014, Sean Farley <sean.michael.farley at gmail.com>
wrote:

>
> bsmith at mcs.anl.gov <javascript:;> writes:
>
> > On Jan 28, 2014, at 12:14 PM, Geoff Oxberry <goxberry at gmail.com<javascript:;>>
> wrote:
> >
> >>
> >>
> >>
> >> On Tue, Jan 28, 2014 at 9:10 AM, Sean Farley <
> sean.michael.farley at gmail.com <javascript:;>> wrote:
> >>
> >> goxberry at gmail.com <javascript:;> writes:
> >>
> >> > To echo what Aron said, I wouldn't point people at the
> >> > hpc.sourceforge.netbuilds. They do install directly into /usr/bin, and
> >> > it's a pain in the ass
> >>
> >> Satish is probably right here about the build location. It's been three
> or four years since I've installed it this way. I stand by that it's still
> difficult to revert. I actually tried this method because of PETSc and
> regretted it because the experience was terrible. Using a package manager
> is more maintainable, and I think PETSc's recommendation of the
> hpc.sourceforge build is a disservice to both users and to PETSc's
> excellent reputation.
> >
> >    I think package managers for Mac OS are a disservice to the community
> and recommend not using them. (See all the discussions in these emails
> about how they fuck up).
>
> Sigh. It is this type of curmudgeon behavior that pushes away people
> from helping out with these type of projects. Packagers are just
> volunteers and to estrange the current three (yes, three) would be
> unfortunate. Not many (read: none) of the other devs care about having
> multiple compilers (thanks, fortran) nor pandering to the scientific
> community's lack of good software practices.
>
>
Agreed.


> It is no secret that MacPorts has historically flubbed on lots of
> PETSc-related issues. I have been trying to change this perspective
> but this email thread pretty succinctly explains what makes my job
> difficult.


Also agreed; the other package managers suffer from this problem as well.


> Just look at how difficult it is to install these packages: superlu,
> superlu_dist, metis, parmetis, scotch, scalapack, and mumps.
>
> The comments here do nothing but drive away users and frustrate
> potential collaborators. Not just for PETSc but for any project that
> depends on PETSc (SLEPc, FEniCS, MOOSE, etc). The true disservice to the
> community is forcing each user to manage their own packages.


Absolutely. Most scientists don't care about these issues until it bites
them in the ass.


> Instead of criticizing here, the energy could be better spent by
> contributing.
>

Couldn't agree more. Had I not had a bad experience with MacPorts, I would
be using SciencePorts, and I think it's important to work on packaging
issues. My biases aside, I hope MacPorts improves, and I think it's good
that you're working on it.


-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/31798823/attachment.html>

From irving at naml.us  Tue Jan 28 18:04:01 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 16:04:01 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
Message-ID: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>

What's the cleanest way to save data from a petsc finite element
simulation including boundary conditions?  First, is it correct that
the local sections contains dofs for boundary conditions, while the
global section does not?  If so, is the following the right plan?

1. At start, save the DMPlex
2. At start, save the local section
3. Every frame, sync the global vector a local vector and write out in
petsc binary format.

Are there standard binary formats for DMPlex and sections?  All I see
is the following sad commented out call to DMPlexView_Binary.

Geoffrey


From balay at mcs.anl.gov  Tue Jan 28 18:16:59 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 18:16:59 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m2a9ef3ckh.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
Message-ID: <alpine.LFD.2.11.1401281805220.18968@asterix>

On Tue, 28 Jan 2014, Sean Farley wrote:

> 
> balay at mcs.anl.gov writes:
> 
> > On Tue, 28 Jan 2014, Sean Farley wrote:
> >
> >> 
> >> mfadams at lbl.gov writes:
> >> 
> >> > Oh yes.  I had a use-make=make commended out. I uncommented it and its
> >> > seems to have gotten MPICH installed.
> >> 
> >> There is a newly (only a few days old) updated mpich port that I now use
> >> with my petsc install.
> >
> > This is not really a fix - as the 'gmake' error will pop up with petsc
> > part of the build aswell.
> 
> False. It works perfectly fine here.

That doesn't mean it woud work for Mark. Previously [a few months
back] Mark had the same issue - and you've responded that you couldn't
reproduce it.

> >> > On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >> >
> >> >> I'll have to check back my e-mail archive - but I think
> >> >> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
> >> >>
> >> >> I thought petsc configure prefred /usr/bin/make - if it was found.
> >> >>
> >> >> One more -ve for macports..
> >> 
> >> There is a fundamental problem with using two stacks. Similar, I think,
> >> to trying to simultaneously link to libc++ and libstdc++. It is just
> >> asking for trouble.
> >
> > I'm not sure how this is causing gmake 'internal' error..
> 
> It doesn't necessarily mean that gmake is broken but could be something
> mismatched. I can't tell without the internal logs.
> 
> Though, I recommend that Mark not use any --download options through
> petsc because those packages are now in MacPorts and mixing two stacks
> is just asking for trouble.

So - unless Mark is also using 'macports' install of PETSc - he is
asking for trouble? [otherwise he can easily end up mixing these
exteranlpackages from macports - and build petsc with native OSX
compilers - so will end up mixing stack anyway?]

Satish


From knepley at gmail.com  Tue Jan 28 18:18:22 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 18:18:22 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
Message-ID: <CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>

On Tue, Jan 28, 2014 at 6:04 PM, Geoffrey Irving <irving at naml.us> wrote:

> What's the cleanest way to save data from a petsc finite element
> simulation including boundary conditions?  First, is it correct that
> the local sections contains dofs for boundary conditions, while the
> global section does not?  If so, is the following the right plan?
>
> 1. At start, save the DMPlex
> 2. At start, save the local section
> 3. Every frame, sync the global vector a local vector and write out in
> petsc binary format.
>
> Are there standard binary formats for DMPlex and sections?  All I see
> is the following sad commented out call to DMPlexView_Binary.


We are having a big discussion about this on petsc-dev. I will write a
longer
thing, but now the only thing that works all the way is VTK. You should just
be able to create the Viewer, do VecView for your fields, and then destroy
it.

I create a global section with all the boundary values in it.

I plan on having HDF5+Xdmf and Exodus.

   Matt


>
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/e277a9d9/attachment.html>

From balay at mcs.anl.gov  Tue Jan 28 18:48:48 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 18:48:48 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1401281818130.18968@asterix>

On Tue, 28 Jan 2014, Geoff Oxberry wrote:

> On Tuesday, January 28, 2014, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Tue, 28 Jan 2014, Barry Smith wrote:
> >
> > > >
> > > > Satish is probably right here about the build location. It's been
> > three or four years since I've installed it this way. I stand by that it's
> > still difficult to revert. I actually tried this method because of PETSc
> > and regretted it because the experience was terrible. Using a package
> > manager is more maintainable, and I think PETSc's recommendation of the
> > hpc.sourceforge build is a disservice to both users and to PETSc's
> > excellent reputation.
> > >
> > >    I think package managers for Mac OS are a disservice to the community
> > and recommend not using them. (See all the discussions in these emails
> > about how they fuck up).
> > >
> >
> > My view is: anyone using OSX has bought into the idea of not having a
> > proper package management system. [yeah you get easy-install packages
> > - but most of them don't have an proper way to uninstall - unless its
> > an "osx-app" which you can drag/drop into trash]
> >
> > gfortran from hpc.sourceforge does things "no worse" than most packages
> > that are available for OSX.
> >
> > Its not obvious - but one can use the file listing from the tarball [as
> > mentioned in my previous e-mail to uninstall]. And is tucked away
> > in /usr/local - so it doesn't do any damage like other packages.
> 
> 
> If you follow the relative install paths, yes, your method does the job.
> It's what I did, with some testing, and then I reinstalled the XCode
> Command Line Tools.

Just to clarify did you have to reinstall xcode because it broke when
you uninstalled hpc.sf gfortran?  [I had to do that due to R's
gfortran - but never for hpc - at it was in /usr/local - and never
interfered with xcode]

> As long as a user is careful, no harm done; it's also
> easy to mess up. You could do something similar via lsbom for packages in
> OS X and/or delete via the package identifier using pkgutil and largely
> only leave behind plist files. This method is what homebrew-cask uses for
> managing OS X binaries.

I agree any package on OSX can be deleted by using the above methond
[instead of quering tarball for file-list - query the pkg database for
file-list]. I've had to do this a couple of times.

On an added note - I've also experimented with installing gfortran from hpc
in say ~/soft/gf/bin/gfortran instead of /usr/local/bin/gfortran so
that uninstall is just 'rm -rf ~/soft/gf/'. This would work after fixing
the dylibs with:
find . -type f -name "*.dylib" -exec install_name_tool -id `pwd`/{} {} \;

[Last time I did this - I had to do some additional stuff - but there
might have been some extra issues with this install]. Since 'tar -tf'
was relatively easier - I've settled on /usr/local/bin/gfortran for
past few years.

> 
> > [for eg: install mercurial for OSX - and see if you can uninstall it]
> 
> 
> For Mercurial, I'd install & uninstall via pip and a virtualenv. Your point
> is well-taken for packages in general, and can still be managed (see
> above). Uninstall apps will also do the job.

Yes there are many ways of installing mercurial. If one were to
install binary from http://mercurial.selenic.com/downloads/ - one
would have to use the 'query pkg database to get file-list and delete
them' approach [as it doesn't have uninstall option].

I refered to this to say 'uninstalling gfortran from hpc is no worse
than uninstalling such packages'. Its slightly better because nuking
/usr/local won't break the core OS files]

> > I agree a better package management system [aka macports/homebrew]
> > should be preferable. But with all the wierd issues that keep comping
> > up with users using macports on petsc lists - I can't convince myself
> > that it is a better recommendation.
> >
> > perhaps homebrew is better - I don't know.
> 
> You guys do the support work (and do a good job of it), so I defer to your
> judgment here. My opinion was borne out of a bad experience and trying
> several methods of installation; I apologize for the brief burst of
> impertinence.
> 
> I would not recommend MacPorts either; I don't use it. Sean knows more
> about this than I do and can better defend MacPorts.
> 
> With Homebrew, it should be possible to replicate the current
> recommendation and only install six additional packages (cloog, gmp, isl,
> mpfr, libmpc, and pkg-config).
> 
> Mixing gfortran 4.8 with the rest of the gcc 4.2 or clang 3.3 stack can
> also be tricky, which was my point about different compiler versions.

Any known issues? gfortran-4.8 from HPC [with apple gcc-4.2] is used
in our default nightly test suite - and we haven't seen any issues
[wrt petsc]

> SciPy
> recommends the AT&T build and states that problems may arise with the
> hpc.sourceforge build; as you pointed out, this version causes problems for
> PETSc. So I tend to use multiple builds of PETSc in different PETSC_ARCH
> directories; one of these builds is a gcc 4.8 build because PETSc is
> relatively self-contained (which is a testament to your design and build
> system), so I'm not terribly worried about system library conflicts.
> 
> My Python rant is not a good argument for hpc.sourceforge (or against
> package managers) because sitewide installs of interpreted language
> packages -- especially via an OS package manager -- should be minimized,
> regardless of operating system or distribution. Otherwise, you run into
> problems like pip trying to overwrite system libraries.
> 
> I would aswell recommend virtualbox with linux as a superior choice.
> >
> > Satish
> >
> 
> Probably the best choice. Even then, some can't run virtual machines (I
> can't at work), and OS X is a lesser evil than a Windows machine.
> 
> As Jed points out, manual installation is difficult to maintain. Fixing
> a package manager is my best path forward (not getting a different job);
> they're not perfect, but they won't get better unless people work on
> them. I agree with Matt that most of this discussion is ideology.

Matt know hows to debug/fix and workarround issues if they come up :)

The primary source of contention here is 'which gfortran' to
recommend.

If folks on petsc lists can support MacPorts and Homebrew - I can
update the FAQ accordingly [with the 3 options listed]

thanks,
Satish


From sean.michael.farley at gmail.com  Tue Jan 28 18:52:43 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 18:52:43 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281805220.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
Message-ID: <m261p338ys.fsf@gmail.com>


balay at mcs.anl.gov writes:

> On Tue, 28 Jan 2014, Sean Farley wrote:
>
>> 
>> balay at mcs.anl.gov writes:
>> 
>> > On Tue, 28 Jan 2014, Sean Farley wrote:
>> >
>> >> 
>> >> mfadams at lbl.gov writes:
>> >> 
>> >> > Oh yes.  I had a use-make=make commended out. I uncommented it and its
>> >> > seems to have gotten MPICH installed.
>> >> 
>> >> There is a newly (only a few days old) updated mpich port that I now use
>> >> with my petsc install.
>> >
>> > This is not really a fix - as the 'gmake' error will pop up with petsc
>> > part of the build aswell.
>> 
>> False. It works perfectly fine here.
>
> That doesn't mean it woud work for Mark. Previously [a few months
> back] Mark had the same issue - and you've responded that you couldn't
> reproduce it.
>
>> >> > On Tue, Jan 28, 2014 at 1:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>> >> >
>> >> >> I'll have to check back my e-mail archive - but I think
>> >> >> '/opt/local/bin/gmake' is broken. Don't you have /usr/bin/make?
>> >> >>
>> >> >> I thought petsc configure prefred /usr/bin/make - if it was found.
>> >> >>
>> >> >> One more -ve for macports..
>> >> 
>> >> There is a fundamental problem with using two stacks. Similar, I think,
>> >> to trying to simultaneously link to libc++ and libstdc++. It is just
>> >> asking for trouble.
>> >
>> > I'm not sure how this is causing gmake 'internal' error..
>> 
>> It doesn't necessarily mean that gmake is broken but could be something
>> mismatched. I can't tell without the internal logs.
>> 
>> Though, I recommend that Mark not use any --download options through
>> petsc because those packages are now in MacPorts and mixing two stacks
>> is just asking for trouble.
>
> So - unless Mark is also using 'macports' install of PETSc - he is
> asking for trouble?

No.

> [otherwise he can easily end up mixing these
> exteranlpackages from macports - and build petsc with native OSX
> compilers - so will end up mixing stack anyway?]

The gmake issue could be a lot of things:

- something weird with mpich's autoconf
- libtool problems
- gmake itself
- path issues

I'll try to reproduce this again to see if it happens on my machine.

Maybe I should just advise not to use --download-*mpi*. Wrapping
compilers is tricky. For example, Mark seems to have upgraded to
Mavericks, yet in his script he is specifying gcc/g++ which is no longer
distributed with Xcode.

For my own work, I install the Apple clang + gfortran 4.8 most of the
time, via

$ sudo port install mpich-default
$ sudo port select --set mpi mpich-mp-fortran

Then try to build PETSc (minus the --download-mpich and CC, CXX, FC
options). You could also try to install most of the external packages:

$ sudo port install hypre ml parmetis metis hdf5-18 netcdf triangle \
superlu superlu_dist mumps

ctetgen and exodusii aren't in MacPorts but (assuming the licenses allow
it) I could add them.


From balay at mcs.anl.gov  Tue Jan 28 19:03:16 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 19:03:16 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m261p338ys.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
	<m261p338ys.fsf@gmail.com>
Message-ID: <alpine.LFD.2.11.1401281854260.18968@asterix>

On Tue, 28 Jan 2014, Sean Farley wrote:

> The gmake issue could be a lot of things:
> 
> - something weird with mpich's autoconf
> - libtool problems
> - gmake itself
> - path issues

The issue is with 'gmake - j 7 all' with mpich build [and similarly
petsc build] after configure [of the corresponding package] is
completed.

> 
> I'll try to reproduce this again to see if it happens on my machine.
> 
> Maybe I should just advise not to use --download-*mpi*. Wrapping
> compilers is tricky. For example, Mark seems to have upgraded to
> Mavericks, yet in his script he is specifying gcc/g++ which is no longer
> distributed with Xcode.

Apple [Xcode5] provides a binary called /usr/bin/gcc - which is
'clang'.  [Yeah naming 'clang' as 'gcc' is funky - but I think apple
does that to not break user makefiles that have 'gcc' hardcoded]

Satish

> 
> For my own work, I install the Apple clang + gfortran 4.8 most of the
> time, via
> 
> $ sudo port install mpich-default
> $ sudo port select --set mpi mpich-mp-fortran
> 
> Then try to build PETSc (minus the --download-mpich and CC, CXX, FC
> options). You could also try to install most of the external packages:
> 
> $ sudo port install hypre ml parmetis metis hdf5-18 netcdf triangle \
> superlu superlu_dist mumps
> 
> ctetgen and exodusii aren't in MacPorts but (assuming the licenses allow
> it) I could add them.
> 



From irving at naml.us  Tue Jan 28 19:03:38 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 17:03:38 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
Message-ID: <CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>

On Tue, Jan 28, 2014 at 4:18 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Jan 28, 2014 at 6:04 PM, Geoffrey Irving <irving at naml.us> wrote:
>>
>> What's the cleanest way to save data from a petsc finite element
>> simulation including boundary conditions?  First, is it correct that
>> the local sections contains dofs for boundary conditions, while the
>> global section does not?  If so, is the following the right plan?
>>
>> 1. At start, save the DMPlex
>> 2. At start, save the local section
>> 3. Every frame, sync the global vector a local vector and write out in
>> petsc binary format.
>>
>> Are there standard binary formats for DMPlex and sections?  All I see
>> is the following sad commented out call to DMPlexView_Binary.
>
> We are having a big discussion about this on petsc-dev. I will write a longer
> thing, but now the only thing that works all the way is VTK. You should just
> be able to create the Viewer, do VecView for your fields, and then destroy
> it.

Thanks, and apologies for missing that thread.

Is there a working example of that kind of viewer use?
DMPlexVTKWriteAll_ASCII seems to traverse a linked list of vectors
attached to the viewer and write them all out, which is a bit
confusing.

> I create a global section with all the boundary values in it.

Do you mean this is what I'll need to do, or is that taken care of
automatically in the VTK path?  It seems like boundary condition
handling can't be automatic, but DMPlexVTKWriteAll_ASCII reads a bunch
of section information out of the DM so I'm sure how one would
substitute a global section + boundaries.

> I plan on having HDF5+Xdmf and Exodus.

Thanks,
Geoffrey


From balay at mcs.anl.gov  Tue Jan 28 19:14:25 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 19:14:25 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m261p338ys.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
	<m261p338ys.fsf@gmail.com>
Message-ID: <alpine.LFD.2.11.1401281907040.18968@asterix>

On Tue, 28 Jan 2014, Sean Farley wrote:

> >> > I'm not sure how this is causing gmake 'internal' error..
> >> 
> >> It doesn't necessarily mean that gmake is broken but could be something
> >> mismatched. I can't tell without the internal logs.
> >> 
> >> Though, I recommend that Mark not use any --download options through
> >> petsc because those packages are now in MacPorts and mixing two stacks
> >> is just asking for trouble.
> >
> > So - unless Mark is also using 'macports' install of PETSc - he is
> > asking for trouble?
> 
> No.

Ok - you mean - "if Mark has superlu_dist installed by macports - and
also uses --download-superlu_dist - then he will have 2 copies of
superlu_dist - and the configure might test for one of the installs
[during configure state] - but the linker can pick up the other one
[based on the order of -L paths added by configure] - and cause wierd
link/runtime errors."

I agree that a problem - and we've seen it before. The fix is to have
only install of such package [either via --download-package or from
macports - but not have both installs]

We've also had this issues with folks having blas/lapack, X11 packages
installed in both system location & macports.

Satish


From knepley at gmail.com  Tue Jan 28 19:27:17 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 19:27:17 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
Message-ID: <CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>

On Tue, Jan 28, 2014 at 7:03 PM, Geoffrey Irving <irving at naml.us> wrote:

> On Tue, Jan 28, 2014 at 4:18 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Jan 28, 2014 at 6:04 PM, Geoffrey Irving <irving at naml.us> wrote:
> >>
> >> What's the cleanest way to save data from a petsc finite element
> >> simulation including boundary conditions?  First, is it correct that
> >> the local sections contains dofs for boundary conditions, while the
> >> global section does not?  If so, is the following the right plan?
> >>
> >> 1. At start, save the DMPlex
> >> 2. At start, save the local section
> >> 3. Every frame, sync the global vector a local vector and write out in
> >> petsc binary format.
> >>
> >> Are there standard binary formats for DMPlex and sections?  All I see
> >> is the following sad commented out call to DMPlexView_Binary.
> >
> > We are having a big discussion about this on petsc-dev. I will write a
> longer
> > thing, but now the only thing that works all the way is VTK. You should
> just
> > be able to create the Viewer, do VecView for your fields, and then
> destroy
> > it.
>
> Thanks, and apologies for missing that thread.
>
> Is there a working example of that kind of viewer use?
> DMPlexVTKWriteAll_ASCII seems to traverse a linked list of vectors
> attached to the viewer and write them all out, which is a bit
> confusing.


In ex62:

  if (user.runType == RUN_FULL) {
    PetscViewer viewer;
    Vec         uLocal;
    const char *name;

    ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
    ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
    ierr = PetscViewerSetFormat(viewer,
PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
    ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);

    ierr = DMGetLocalVector(dm, &uLocal);CHKERRQ(ierr);
    ierr = PetscObjectGetName((PetscObject) u, &name);CHKERRQ(ierr);
    ierr = PetscObjectSetName((PetscObject) uLocal, name);CHKERRQ(ierr);
    ierr = DMGlobalToLocalBegin(dm, u, INSERT_VALUES, uLocal);CHKERRQ(ierr);
    ierr = DMGlobalToLocalEnd(dm, u, INSERT_VALUES, uLocal);CHKERRQ(ierr);
    ierr = VecView(uLocal, viewer);CHKERRQ(ierr);
    ierr = DMRestoreLocalVector(dm, &uLocal);CHKERRQ(ierr);

    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
  }



> > I create a global section with all the boundary values in it.
>
> Do you mean this is what I'll need to do, or is that taken care of
> automatically in the VTK path?  It seems like boundary condition
> handling can't be automatic, but DMPlexVTKWriteAll_ASCII reads a bunch
> of section information out of the DM so I'm sure how one would
> substitute a global section + boundaries.


You are right, this is not what I do anymore. I decided it was easier for
all the Viewers
to just take local vectors. I handle all the "globalization" automatically.
See the code snippet above.

  Thanks,

    Matt


> > I plan on having HDF5+Xdmf and Exodus.
>
> Thanks,
> Geoffrey
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/2f826ca0/attachment.html>

From sean.michael.farley at gmail.com  Tue Jan 28 19:34:23 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 19:34:23 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281854260.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
	<m261p338ys.fsf@gmail.com> <alpine.LFD.2.11.1401281854260.1
	8968@asterix>
Message-ID: <m21tzr371c.fsf@gmail.com>


balay at mcs.anl.gov writes:

> On Tue, 28 Jan 2014, Sean Farley wrote:
>
>> The gmake issue could be a lot of things:
>> 
>> - something weird with mpich's autoconf
>> - libtool problems
>> - gmake itself
>> - path issues
>
> The issue is with 'gmake - j 7 all' with mpich build [and similarly
> petsc build] after configure [of the corresponding package] is
> completed.

So I see. I still can't reproduce but it seems that other gmake users
have this issue on case-insensitive filesystems (which I don't
have). Can you reproduce this on a vm easily? It'd be great to nail this
down.

>> I'll try to reproduce this again to see if it happens on my machine.
>> 
>> Maybe I should just advise not to use --download-*mpi*. Wrapping
>> compilers is tricky. For example, Mark seems to have upgraded to
>> Mavericks, yet in his script he is specifying gcc/g++ which is no longer
>> distributed with Xcode.
>
> Apple [Xcode5] provides a binary called /usr/bin/gcc - which is
> 'clang'.  [Yeah naming 'clang' as 'gcc' is funky - but I think apple
> does that to not break user makefiles that have 'gcc' hardcoded]

Ah, right, I forgot. Confusing, nonetheless.


From sean.michael.farley at gmail.com  Tue Jan 28 19:35:43 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Tue, 28 Jan 2014 19:35:43 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281907040.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
	<m261p338ys.fsf@gmail.com> <alpine.LFD.2.11.1401281907040.1
	8968@asterix>
Message-ID: <m2zjmf1seo.fsf@gmail.com>


balay at mcs.anl.gov writes:

> On Tue, 28 Jan 2014, Sean Farley wrote:
>
>> >> > I'm not sure how this is causing gmake 'internal' error..
>> >> 
>> >> It doesn't necessarily mean that gmake is broken but could be something
>> >> mismatched. I can't tell without the internal logs.
>> >> 
>> >> Though, I recommend that Mark not use any --download options through
>> >> petsc because those packages are now in MacPorts and mixing two stacks
>> >> is just asking for trouble.
>> >
>> > So - unless Mark is also using 'macports' install of PETSc - he is
>> > asking for trouble?
>> 
>> No.
>
> Ok - you mean - "if Mark has superlu_dist installed by macports - and
> also uses --download-superlu_dist - then he will have 2 copies of
> superlu_dist - and the configure might test for one of the installs
> [during configure state] - but the linker can pick up the other one
> [based on the order of -L paths added by configure] - and cause wierd
> link/runtime errors."
>
> I agree that a problem - and we've seen it before. The fix is to have
> only install of such package [either via --download-package or from
> macports - but not have both installs]

Yes, this is no doubt a common problem.

> We've also had this issues with folks having blas/lapack, X11 packages
> installed in both system location & macports.

Yes, ATLAS being installed is mostly a nightmare. I now have it
installed solely to fix linking issues.


From jed at jedbrown.org  Tue Jan 28 19:55:38 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 19:55:38 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAPhiW4gx-ckvAZfDjCAqGDtPaQikj6nX83yaGBrmuAo1i5183Q@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
	<CAPhiW4gx-ckvAZfDjCAqGDtPaQikj6nX83ya GBrmuAo1
	i5183Q@mail.gmail.com>
Message-ID: <87d2jbmu05.fsf@jedbrown.org>

Aron Ahmadia <aron at ahmadia.net> writes:

> FWIW, on OS X.9, the venerable make 3.81 is installed natively and does not
> seem to cause much grief, despite its age.

That's good to hear.  Why do they refuse to ship a make that is less
than 8 years old?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/b1b408fe/attachment.sig>

From balay at mcs.anl.gov  Tue Jan 28 20:19:01 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 20:19:01 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87d2jbmu05.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
	<CAPhiW4gx-ckvAZfDjCAqGDtPaQikj6nX83ya GBrmuAo1
	i5183Q@mail.gmail.com> <87d2jbmu05.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401282014090.18968@asterix>

On Tue, 28 Jan 2014, Jed Brown wrote:

> Aron Ahmadia <aron at ahmadia.net> writes:
> 
> > FWIW, on OS X.9, the venerable make 3.81 is installed natively and does not
> > seem to cause much grief, despite its age.
> 
> That's good to hear.  Why do they refuse to ship a make that is less
> than 8 years old?

Our linux compute machines [ubuntu 12.04] is also has  3.81

Fedora Linux is still at 3.82
http://koji.fedoraproject.org/koji/packageinfo?packageID=813

Per https://ia601008.us.archive.org/7/items/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3
Apple doesn't want touch GPL3 code [hence froze gcc at 4.2 and started investing in clang]. But this
issue doesn't exist for gnumake. [I see its GPL2+]

Satish


From jed at jedbrown.org  Tue Jan 28 20:41:08 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 20:41:08 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401282014090.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
	<87d2jbmu05.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401282014090.18968@asterix>
Message-ID: <87a9efmrwb.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> On Tue, 28 Jan 2014, Jed Brown wrote:
>
>> That's good to hear.  Why do they refuse to ship a make that is less
>> than 8 years old?
>
> Our linux compute machines [ubuntu 12.04] is also has  3.81
>
> Fedora Linux is still at 3.82
> http://koji.fedoraproject.org/koji/packageinfo?packageID=813

4.0 was only released last fall, but 3.82 has been out for several
years.  It had a very minor backward incompatibility in resolving
multiple applicable pattern rules (new rule is that the shortest stem
wins) which affected a couple packages at the time, though they promply
updated.

> Per https://ia601008.us.archive.org/7/items/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3
> Apple doesn't want touch GPL3 code [hence froze gcc at 4.2 and started
> investing in clang]. 

They hired Chris Lattner in 2005, well before GPLv3 (2007).

> But this issue doesn't exist for gnumake. [I see its GPL2+]

No, it is GPLv3+ since 3.82.  Stupid ideology wars and the only true
losers are the users and developers caught in the middle.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/e110ff92/attachment.sig>

From balay at mcs.anl.gov  Tue Jan 28 20:55:16 2014
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 28 Jan 2014 20:55:16 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87a9efmrwb.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
	<87d2jbmu05.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401282014090.18968@asterix>
	<87a9efmrwb.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1401282044200.18968@asterix>

On Tue, 28 Jan 2014, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > On Tue, 28 Jan 2014, Jed Brown wrote:
> >
> >> That's good to hear.  Why do they refuse to ship a make that is less
> >> than 8 years old?
> >
> > Our linux compute machines [ubuntu 12.04] is also has  3.81
> >
> > Fedora Linux is still at 3.82
> > http://koji.fedoraproject.org/koji/packageinfo?packageID=813
> 
> 4.0 was only released last fall, but 3.82 has been out for several
> years.  It had a very minor backward incompatibility in resolving
> multiple applicable pattern rules (new rule is that the shortest stem
> wins) which affected a couple packages at the time, though they promply
> updated.

For one - 4.0 its not yet in fedora rawhide [I don't know why]. Also
fedora appears to carry lot of patches for 3.82.
http://lwn.net/Articles/569841/

Perhaps some of them are related to this issue on OSX?

> > Per https://ia601008.us.archive.org/7/items/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3
> > Apple doesn't want touch GPL3 code [hence froze gcc at 4.2 and started
> > investing in clang]. 
> 
> They hired Chris Lattner in 2005, well before GPLv3 (2007).

GPLv3 was in development for many years. So I guess apple planned ahead.
http://en.wikipedia.org/wiki/Gpl_v3#Version_3

> > But this issue doesn't exist for gnumake. [I see its GPL2+]
> 
> No, it is GPLv3+ since 3.82.  Stupid ideology wars and the only true
> losers are the users and developers caught in the middle.

Ok - so GPLv3 must be the reason Apple is sticking with gnumake 3.81

Fedora is labeling it as GPLv2+ - so I misread that
$ rpm -qi make |grep License
License     : GPLv2+

Satish



From jed at jedbrown.org  Tue Jan 28 21:33:11 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 21:33:11 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401282044200.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<87fvo7naac.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401281405470.18968@asterix>
	<alpine.LFD.2.11.1401281410300.18968@asterix>
	<87d2jbmu05.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401282014090.18968@asterix>
	<87a9efmrwb.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401282044200.18968@asterix>
Message-ID: <877g9jmphk.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

>> They hired Chris Lattner in 2005, well before GPLv3 (2007).
>
> GPLv3 was in development for many years. So I guess apple planned ahead.

They always hated every kind of GPL.  Lattner had a vision and built
something worthwhile.  Two years after the first release, Apple decided
it was promising enough to hire him.  GPLv3 was just a place they
decided to draw the line, and unfortunately, it has given many of their
users a distorted impression about the quality/age of products like GCC.

> Fedora is labeling it as GPLv2+ - so I misread that
> $ rpm -qi make |grep License
> License     : GPLv2+

File a bug report.  I would expect Fedora to be careful about this.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/af14b613/attachment.sig>

From jed at jedbrown.org  Tue Jan 28 21:41:55 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 21:41:55 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
Message-ID: <874n4nmp30.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> In ex62:
>
>   if (user.runType == RUN_FULL) {
>     PetscViewer viewer;
>     Vec         uLocal;
>     const char *name;
>
>     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
>     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
>     ierr = PetscViewerSetFormat(viewer,
> PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);

Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
name to some *.vtu; the VTK viewer will infer output format from
extension and write the VTU XML format with binary-appended data in a
fairly fast and memory-scalable way.  Then

  VecView(u,viewer);


>     ierr = DMGetLocalVector(dm, &uLocal);CHKERRQ(ierr);
>     ierr = PetscObjectGetName((PetscObject) u, &name);CHKERRQ(ierr);
>     ierr = PetscObjectSetName((PetscObject) uLocal, name);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalBegin(dm, u, INSERT_VALUES, uLocal);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalEnd(dm, u, INSERT_VALUES, uLocal);CHKERRQ(ierr);
>     ierr = VecView(uLocal, viewer);CHKERRQ(ierr);
>     ierr = DMRestoreLocalVector(dm, &uLocal);CHKERRQ(ierr);

None of the above, then close the file:

>     ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>   }

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/c84b65cf/attachment.sig>

From knepley at gmail.com  Tue Jan 28 21:49:40 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 21:49:40 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <874n4nmp30.fsf@jedbrown.org>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
Message-ID: <CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>

On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > In ex62:
> >
> >   if (user.runType == RUN_FULL) {
> >     PetscViewer viewer;
> >     Vec         uLocal;
> >     const char *name;
> >
> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
> >     ierr = PetscViewerSetFormat(viewer,
> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
> >     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);
>
> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
> name to some *.vtu; the VTK viewer will infer output format from
> extension and write the VTU XML format with binary-appended data in a
> fairly fast and memory-scalable way.  Then
>
>   VecView(u,viewer);


I think Jed is wrong here. You need to be using local vectors since they
have
BC and constraints in them. Please explain how your VTU format will
magically
put them in.

   Matt


> >     ierr = DMGetLocalVector(dm, &uLocal);CHKERRQ(ierr);
> >     ierr = PetscObjectGetName((PetscObject) u, &name);CHKERRQ(ierr);
> >     ierr = PetscObjectSetName((PetscObject) uLocal, name);CHKERRQ(ierr);
> >     ierr = DMGlobalToLocalBegin(dm, u, INSERT_VALUES,
> uLocal);CHKERRQ(ierr);
> >     ierr = DMGlobalToLocalEnd(dm, u, INSERT_VALUES,
> uLocal);CHKERRQ(ierr);
> >     ierr = VecView(uLocal, viewer);CHKERRQ(ierr);
> >     ierr = DMRestoreLocalVector(dm, &uLocal);CHKERRQ(ierr);
>
> None of the above, then close the file:
>
> >     ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
> >   }
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/41a591f9/attachment.html>

From jed at jedbrown.org  Tue Jan 28 22:24:57 2014
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 28 Jan 2014 22:24:57 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
Message-ID: <871tzrmn3a.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>> > In ex62:
>> >
>> >   if (user.runType == RUN_FULL) {
>> >     PetscViewer viewer;
>> >     Vec         uLocal;
>> >     const char *name;
>> >
>> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
>> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
>> >     ierr = PetscViewerSetFormat(viewer,
>> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>> >     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);
>>
>> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
>> name to some *.vtu; the VTK viewer will infer output format from
>> extension and write the VTU XML format with binary-appended data in a
>> fairly fast and memory-scalable way.  Then
>>
>>   VecView(u,viewer);
>
>
> I think Jed is wrong here. You need to be using local vectors since
> they have BC and constraints in them. Please explain how your VTU
> format will magically put them in.

We made this work when I threw a temper tantrum while we were writing TS
ex11.c (late 2012).

PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
{
  DM             dm;
  PetscBool      isvtk;
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
  if (!dm) SETERRQ(PetscObjectComm((PetscObject)v), PETSC_ERR_ARG_WRONG, "Vector not generated from a DM");
  ierr = PetscObjectTypeCompare((PetscObject) viewer, PETSCVIEWERVTK, &isvtk);CHKERRQ(ierr);
  if (isvtk) {
    Vec         locv;
    const char *name;

    ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
    ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
    ierr = PetscObjectSetName((PetscObject) locv, name);CHKERRQ(ierr);
    ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
    ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
    ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
    ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/bdfaa6f8/attachment.sig>

From knepley at gmail.com  Tue Jan 28 22:40:23 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 28 Jan 2014 22:40:23 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <871tzrmn3a.fsf@jedbrown.org>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
Message-ID: <CAMYG4GnXJCcomcpjmvVch5RTsuBPV3QEebWVo71En8QtrN5qCQ@mail.gmail.com>

That's right. We need to change all the examples.

   Matt
On Jan 28, 2014 10:25 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Matthew Knepley <knepley at gmail.com> writes:
> >> > In ex62:
> >> >
> >> >   if (user.runType == RUN_FULL) {
> >> >     PetscViewer viewer;
> >> >     Vec         uLocal;
> >> >     const char *name;
> >> >
> >> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
> >> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
> >> >     ierr = PetscViewerSetFormat(viewer,
> >> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
> >> >     ierr = PetscViewerFileSetName(viewer,
> "ex62_sol.vtk");CHKERRQ(ierr);
> >>
> >> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
> >> name to some *.vtu; the VTK viewer will infer output format from
> >> extension and write the VTU XML format with binary-appended data in a
> >> fairly fast and memory-scalable way.  Then
> >>
> >>   VecView(u,viewer);
> >
> >
> > I think Jed is wrong here. You need to be using local vectors since
> > they have BC and constraints in them. Please explain how your VTU
> > format will magically put them in.
>
> We made this work when I threw a temper tantrum while we were writing TS
> ex11.c (late 2012).
>
> PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
> {
>   DM             dm;
>   PetscBool      isvtk;
>   PetscErrorCode ierr;
>
>   PetscFunctionBegin;
>   ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
>   if (!dm) SETERRQ(PetscObjectComm((PetscObject)v), PETSC_ERR_ARG_WRONG,
> "Vector not generated from a DM");
>   ierr = PetscObjectTypeCompare((PetscObject) viewer, PETSCVIEWERVTK,
> &isvtk);CHKERRQ(ierr);
>   if (isvtk) {
>     Vec         locv;
>     const char *name;
>
>     ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
>     ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
>     ierr = PetscObjectSetName((PetscObject) locv, name);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>     ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
>     ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/fe021d51/attachment.html>

From irving at naml.us  Tue Jan 28 22:57:56 2014
From: irving at naml.us (Geoffrey Irving)
Date: Tue, 28 Jan 2014 20:57:56 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <871tzrmn3a.fsf@jedbrown.org>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
Message-ID: <CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>

On Tue, Jan 28, 2014 at 8:24 PM, Jed Brown <jed at jedbrown.org> wrote:
> Matthew Knepley <knepley at gmail.com> writes:
>
>> On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Matthew Knepley <knepley at gmail.com> writes:
>>> > In ex62:
>>> >
>>> >   if (user.runType == RUN_FULL) {
>>> >     PetscViewer viewer;
>>> >     Vec         uLocal;
>>> >     const char *name;
>>> >
>>> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
>>> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
>>> >     ierr = PetscViewerSetFormat(viewer,
>>> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>>> >     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);
>>>
>>> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
>>> name to some *.vtu; the VTK viewer will infer output format from
>>> extension and write the VTU XML format with binary-appended data in a
>>> fairly fast and memory-scalable way.  Then
>>>
>>>   VecView(u,viewer);
>>
>> I think Jed is wrong here. You need to be using local vectors since
>> they have BC and constraints in them. Please explain how your VTU
>> format will magically put them in.
>
> We made this work when I threw a temper tantrum while we were writing TS
> ex11.c (late 2012).
>
> PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
> {
>   DM             dm;
>   PetscBool      isvtk;
>   PetscErrorCode ierr;
>
>   PetscFunctionBegin;
>   ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
>   if (!dm) SETERRQ(PetscObjectComm((PetscObject)v), PETSC_ERR_ARG_WRONG, "Vector not generated from a DM");
>   ierr = PetscObjectTypeCompare((PetscObject) viewer, PETSCVIEWERVTK, &isvtk);CHKERRQ(ierr);
>   if (isvtk) {
>     Vec         locv;
>     const char *name;
>
>     ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
>     ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
>     ierr = PetscObjectSetName((PetscObject) locv, name);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>     ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>     ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
>     ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);

Which of those lines adds the correct boundary condition values?

Geoffrey


From asmund.ervik at ntnu.no  Wed Jan 29 01:44:21 2014
From: asmund.ervik at ntnu.no (=?iso-8859-1?Q?=C5smund_Ervik?=)
Date: Wed, 29 Jan 2014 07:44:21 +0000
Subject: [petsc-dev] configure failed after update of OSX
Message-ID: <0E576811AB298343AC632BBCAAEFC3794579C7A8@WAREHOUSE08.win.ntnu.no>

>From: Satish Balay <balay at mcs.anl.gov>
>To: Jed Brown <jed at jedbrown.org>
>Cc: petsc-dev <petsc-dev at mcs.anl.gov>
>Subject: Re: [petsc-dev] configure failed after update of OSX
>
>Our linux compute machines [ubuntu 12.04] is also has  3.81
>
>Fedora Linux is still at 3.82
>http://koji.fedoraproject.org/koji/packageinfo?packageID=813

ArchLinux (which is bleeding edge and rolling release) only upgraded to 4.0 in October. The latest OpenSuse ships 3.82.

And I must say after this lengthy thread on OSX configure calamities, it seems that people doing scientific computing on OSX are doing themselves a disservice. Surely taking half a day up front to install a sane OS is better than spending half a day fixing stuff every time Apple decides to change something?

Just my $0.02 .

Regards,
?smund

From goxberry at gmail.com  Wed Jan 29 02:22:20 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Wed, 29 Jan 2014 00:22:20 -0800
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <alpine.LFD.2.11.1401281818130.18968@asterix>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
Message-ID: <CAHC8rGuid+4Bwbxf_Be6bEdvh9Rm_tC0ew1LiFe1zmuvHbswXQ@mail.gmail.com>

On Tue, Jan 28, 2014 at 4:48 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 28 Jan 2014, Geoff Oxberry wrote:
>
> > On Tuesday, January 28, 2014, Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > On Tue, 28 Jan 2014, Barry Smith wrote:
> > >
> > > > >
> > > > > Satish is probably right here about the build location. It's been
> > > three or four years since I've installed it this way. I stand by that
> it's
> > > still difficult to revert. I actually tried this method because of
> PETSc
> > > and regretted it because the experience was terrible. Using a package
> > > manager is more maintainable, and I think PETSc's recommendation of the
> > > hpc.sourceforge build is a disservice to both users and to PETSc's
> > > excellent reputation.
> > > >
> > > >    I think package managers for Mac OS are a disservice to the
> community
> > > and recommend not using them. (See all the discussions in these emails
> > > about how they fuck up).
> > > >
> > >
> > > My view is: anyone using OSX has bought into the idea of not having a
> > > proper package management system. [yeah you get easy-install packages
> > > - but most of them don't have an proper way to uninstall - unless its
> > > an "osx-app" which you can drag/drop into trash]
> > >
> > > gfortran from hpc.sourceforge does things "no worse" than most packages
> > > that are available for OSX.
> > >
> > > Its not obvious - but one can use the file listing from the tarball [as
> > > mentioned in my previous e-mail to uninstall]. And is tucked away
> > > in /usr/local - so it doesn't do any damage like other packages.
> >
> >
> > If you follow the relative install paths, yes, your method does the job.
> > It's what I did, with some testing, and then I reinstalled the XCode
> > Command Line Tools.
>
> Just to clarify did you have to reinstall xcode because it broke when
> you uninstalled hpc.sf gfortran?  [I had to do that due to R's
> gfortran - but never for hpc - at it was in /usr/local - and never
> interfered with xcode]
>

I did that because I'm paranoid, and there are reports of the hpc.sf
language standard libraries interfering with the system language standard
libraries. I think you're right that it is unnecessary. The AT&T binaries
do require it.


>
> > As long as a user is careful, no harm done; it's also
> > easy to mess up. You could do something similar via lsbom for packages in
> > OS X and/or delete via the package identifier using pkgutil and largely
> > only leave behind plist files. This method is what homebrew-cask uses for
> > managing OS X binaries.
>
> I agree any package on OSX can be deleted by using the above methond
> [instead of quering tarball for file-list - query the pkg database for
> file-list]. I've had to do this a couple of times.
>
> On an added note - I've also experimented with installing gfortran from hpc
> in say ~/soft/gf/bin/gfortran instead of /usr/local/bin/gfortran so
> that uninstall is just 'rm -rf ~/soft/gf/'. This would work after fixing
> the dylibs with:
> find . -type f -name "*.dylib" -exec install_name_tool -id `pwd`/{} {} \;
>
> [Last time I did this - I had to do some additional stuff - but there
> might have been some extra issues with this install]. Since 'tar -tf'
> was relatively easier - I've settled on /usr/local/bin/gfortran for
> past few years.
>
>
The symlinking is more or less what Homebrew does if you install
apple-gcc42. GNU Stow is also supposed to use symlink farms, and I've come
to appreciate the approach precisely because you can delete directories
without much hassle.


> >
> > > [for eg: install mercurial for OSX - and see if you can uninstall it]
> >
> >
> > For Mercurial, I'd install & uninstall via pip and a virtualenv. Your
> point
> > is well-taken for packages in general, and can still be managed (see
> > above). Uninstall apps will also do the job.
>
> Yes there are many ways of installing mercurial. If one were to
> install binary from http://mercurial.selenic.com/downloads/ - one
> would have to use the 'query pkg database to get file-list and delete
> them' approach [as it doesn't have uninstall option].
>
> I refered to this to say 'uninstalling gfortran from hpc is no worse
> than uninstalling such packages'. Its slightly better because nuking
> /usr/local won't break the core OS files]


> > > I agree a better package management system [aka macports/homebrew]
> > > should be preferable. But with all the wierd issues that keep comping
> > > up with users using macports on petsc lists - I can't convince myself
> > > that it is a better recommendation.
> > >
> > > perhaps homebrew is better - I don't know.
> >
> > You guys do the support work (and do a good job of it), so I defer to
> your
> > judgment here. My opinion was borne out of a bad experience and trying
> > several methods of installation; I apologize for the brief burst of
> > impertinence.
> >
> > I would not recommend MacPorts either; I don't use it. Sean knows more
> > about this than I do and can better defend MacPorts.
> >
> > With Homebrew, it should be possible to replicate the current
> > recommendation and only install six additional packages (cloog, gmp, isl,
> > mpfr, libmpc, and pkg-config).
> >
> > Mixing gfortran 4.8 with the rest of the gcc 4.2 or clang 3.3 stack can
> > also be tricky, which was my point about different compiler versions.
>
> Any known issues? gfortran-4.8 from HPC [with apple gcc-4.2] is used
> in our default nightly test suite - and we haven't seen any issues
> [wrt petsc]


Aside from SciPy's warning, no. I wrote a couple cross-language programs to
test it a few years ago, and I'm pretty sure I screwed up the linking when
the two versions were different. If I get some time, I'll retry it and let
you know if something weird happens. There isn't a lot of easily
Google-able material on mixing gfortran from one GCC version and the rest
of the GCC suite from another version; I'll see if I can convince Ondrej
Certik to write docs about building cross-language Fortran code (and not
just writing it). I rebuilt PETSc today on another machine with
gfortran-4.8 and clang-3.3 and it worked fine.


> > SciPy
> > recommends the AT&T build and states that problems may arise with the
> > hpc.sourceforge build; as you pointed out, this version causes problems
> for
> > PETSc. So I tend to use multiple builds of PETSc in different PETSC_ARCH
> > directories; one of these builds is a gcc 4.8 build because PETSc is
> > relatively self-contained (which is a testament to your design and build
> > system), so I'm not terribly worried about system library conflicts.
> >
> > My Python rant is not a good argument for hpc.sourceforge (or against
> > package managers) because sitewide installs of interpreted language
> > packages -- especially via an OS package manager -- should be minimized,
> > regardless of operating system or distribution. Otherwise, you run into
> > problems like pip trying to overwrite system libraries.
> >
> > I would aswell recommend virtualbox with linux as a superior choice.
> > >
> > > Satish
> > >
> >
> > Probably the best choice. Even then, some can't run virtual machines (I
> > can't at work), and OS X is a lesser evil than a Windows machine.
> >
> > As Jed points out, manual installation is difficult to maintain. Fixing
> > a package manager is my best path forward (not getting a different job);
> > they're not perfect, but they won't get better unless people work on
> > them. I agree with Matt that most of this discussion is ideology.
>
> Matt know hows to debug/fix and workarround issues if they come up :)
>
> The primary source of contention here is 'which gfortran' to
> recommend.
>
> If folks on petsc lists can support MacPorts and Homebrew - I can
> update the FAQ accordingly [with the 3 options listed]
>

I can answer some Homebrew questions. I typically skim petsc-dev and
petsc-users due to my relative PETSc inexperience, and only read a few
posts here and there. If someone bugs you about it and I miss it, ping me,
and I'll do my best to help. PETSc is also a package in the
homebrew-science repo, so I can lean on some of them for help also. I've
learned to build PETSc from source because the package builds in Linux
distros and Mac package managers usually lack the external package
capabilities I want (and they're old versions, too).


>
> thanks,
> Satish
>

Cheers,

Geoff

-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/e7326de1/attachment.html>

From goxberry at gmail.com  Wed Jan 29 03:16:25 2014
From: goxberry at gmail.com (Geoff Oxberry)
Date: Wed, 29 Jan 2014 01:16:25 -0800
Subject: [petsc-dev] configure failed after update of OSX
Message-ID: <CAHC8rGtm73vnU7TEhvG5V70ykSbwMxHetTfaAAZArXjtO1dCsA@mail.gmail.com>

>
> Message: 4
> Date: Wed, 29 Jan 2014 07:44:21 +0000
> From: ?smund Ervik <asmund.ervik at ntnu.no>
> To: "petsc-dev at mcs.anl.gov" <petsc-dev at mcs.anl.gov>
> Subject: Re: [petsc-dev] configure failed after update of OSX
> Message-ID:
>         <0E576811AB298343AC632BBCAAEFC3794579C7A8 at WAREHOUSE08.win.ntnu.no>
> Content-Type: text/plain; charset="iso-8859-1"
>
> >From: Satish Balay <balay at mcs.anl.gov>
> >To: Jed Brown <jed at jedbrown.org>
> >Cc: petsc-dev <petsc-dev at mcs.anl.gov>
> >Subject: Re: [petsc-dev] configure failed after update of OSX
> >
> >Our linux compute machines [ubuntu 12.04] is also has  3.81
> >
> >Fedora Linux is still at 3.82
> >http://koji.fedoraproject.org/koji/packageinfo?packageID=813
>
> ArchLinux (which is bleeding edge and rolling release) only upgraded to
> 4.0 in October. The latest OpenSuse ships 3.82.
>
> And I must say after this lengthy thread on OSX configure calamities, it
> seems that people doing scientific computing on OSX are doing themselves a
> disservice. Surely taking half a day up front to install a sane OS is
> better than spending half a day fixing stuff every time Apple decides to
> change something?
>

That's not an option for some people (for instance, work-provided laptops).

What constitutes "a sane OS" is different things to different people. Linux
is great for development, and works very well as an every day operating
system. I use it on one of my work desktops. I would love it if the
compatibility of Microsoft Office with comparable Linux software suites
were better, but it's not, and it makes having to deal with certain parties
(journals, employers, conferences) that insist on proprietary formats
difficult. Asking users to not deal with those organizations may not be
practical.

Choice of hardware can also be a factor. Hardware drivers can be a tricky
thing to worry about with Linux, particularly with laptops. Wireless
support was a big problem, and support has gotten better over time. Kernel
upgrades can wreak havoc on things like nVidia graphics drivers. It can be
tough to locate drivers for peripherals and install them without time,
effort, and expert knowledge. (Currently a problem with my Linux desktop
and the office printer.)

Consumer watchdog agencies (Consumer Reports, for instance) rate Apple high
in customer service, and other companies rate somewhat lower. This aspect
can be important if you want to outsource laptop repair (as I do, because I
prefer to spend my time on other things). In terms of general consumer
usage, Apple laptops rank consistently well also, which may explain why
computational scientists are willing to put up with the rough edges when it
comes to development environments. At a computational science conference
I've attended every year for the past 7 years, the shift to Macs has been
dramatic.

Linus Torvalds was using a MacBook Air for a while; of course, he put Linux
on it.

I like Linux, and I intend to keep using it alongside OS X. Like I said,
it's a great operating system. Ideology aside, I find the arguments against
OS X that can be paraphrased as "you should just use Linux because it has a
great package manager" unconvincing because the proposed solutions they
offer to some of the issues I raise are uncompelling (and they're not the
fault of Linux, either).


> Just my $0.02 .
>
> Regards,
> ?smund
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/ff82324a/attachment.html>

From jed at jedbrown.org  Wed Jan 29 06:10:58 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 06:10:58 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <0E576811AB298343AC632BBCAAEFC3794579C7A8@WAREHOUSE08.win.ntnu.no>
References: <0E576811AB298343AC632BBCAAEFC3794579C7A8@WAREHOUSE08.win.ntnu.no>
Message-ID: <87wqhjkmy5.fsf@jedbrown.org>

[You should be aware that your message didn't set In-Reply-To or
References, thus doesn't thread correctly.  Please check your mailer
settings.]

?smund Ervik <asmund.ervik at ntnu.no> writes:

>>From: Satish Balay <balay at mcs.anl.gov>
>>To: Jed Brown <jed at jedbrown.org>
>>Cc: petsc-dev <petsc-dev at mcs.anl.gov>
>>Subject: Re: [petsc-dev] configure failed after update of OSX
>>
>>Our linux compute machines [ubuntu 12.04] is also has  3.81
>>
>>Fedora Linux is still at 3.82
>>http://koji.fedoraproject.org/koji/packageinfo?packageID=813
>
> ArchLinux (which is bleeding edge and rolling release) only upgraded
> to 4.0 in October. 

4.0 was released Oct 9; I don't blame any system for not having upgraded
yet.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/fd8a5c73/attachment.sig>

From rupp at mcs.anl.gov  Wed Jan 29 07:28:31 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 29 Jan 2014 14:28:31 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E6D43F.9020501@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>	<52E37A65.9090106@mcs.anl.gov>	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>	<52E4293B.3080005@mcs.anl.gov>	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>	<52E4E29B.9080102@mcs.anl.gov>	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>	<52E58EE9.1010001@mcs.anl.gov>	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>	<52E6BC1C.7090701@mcs.anl.gov>
	<877g9lnnvb.fsf@jedbrown.org> <52E6D43F.9020501@mcs.anl.gov>
Message-ID: <52E901FF.6030202@mcs.anl.gov>

Hey,

>>> Mani's build included
>>> --with-threadcomm --with-pthreadclasses --with-openmp
>>> which seems the be the cause of the problem. Without these flags,  the
>>> problem disappears and results are correct. If I remember correctly,
>>> this is a more fundamental problem in threadcomm rather than specific to
>>> the ViennaCL bindings, yet we clearly need to fix it.
>>
>> [repost from petsc-maint]
>>
>> Evidently threads are a liability right now.  Do you know what caused
>> this behavior so we can avoid it in the future?
>
> Unfortunately I don't know what exactly is causing the problem. My best
> guess is that one of the sys-calls inside threadcomm is in conflict with
> the internals of the OpenCL runtime. I'll see whether I can reproduce
> this on my machine, then I can incrementally disable parts of threadcomm
> until I found the cause.

Okay, I could reproduce this problem on my machine: A configuration 
including
  --with-threadcomm --with-pthreadclasses --with-openmp
fails, while
  --with-threadcomm --with-pthreadclasses
works. Note that all runs are single-threaded, so it is only the 
presence of -fopenmp which screws things up. The most immediate 'fix' is 
to prevent the combination of OpenMP and OpenCL at the configure stage. 
The problem does not show up in a fresh build with CUDA and CUSP, at 
least not when using the standard ComputeResidual() based on VecGetArray().

Best regards,
Karli



From jed at jedbrown.org  Wed Jan 29 06:26:35 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 06:26:35 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CAHC8rGuid+4Bwbxf_Be6bEdvh9Rm_tC0ew1LiFe1zmuvHbswXQ@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
	<CAHC8rGuid+4Bwbxf_Be6bEdvh9Rm_tC0ew1 LiFe1zmu
	vHbswXQ@mail.gmail.com>
Message-ID: <87txcnkm84.fsf@jedbrown.org>

Geoff Oxberry <goxberry at gmail.com> writes:
> I can answer some Homebrew questions. I typically skim petsc-dev and
> petsc-users due to my relative PETSc inexperience, and only read a few
> posts here and there. If someone bugs you about it and I miss it, ping me,
> and I'll do my best to help. PETSc is also a package in the
> homebrew-science repo, so I can lean on some of them for help also. I've
> learned to build PETSc from source because the package builds in Linux
> distros and Mac package managers usually lack the external package
> capabilities I want (and they're old versions, too).

For systems with shared libraries, it may be useful to have a way to add
plugins after building PETSc, rather than asking packagers to build
every subset or a maximal subset.  The problem is that functions like
PCHYPRESetType() are made visible to the user and thus become link-time
dependencies.  Is there something *maintainable* we could do to allow
applications to link without the packages?

Functions like the above either do PetscTryMethod or PetscUseMethod so
they don't have a link-time dependency on the implementation
PCHYPRESetType_HYPRE (and thus the external package), but are currently
placed in the file with the external package.  If we moved them
somewhere that was always compiled, we would get rid of the link-time
dependency and then we could add these plugins after installing the
PETSc base system.  It's not a huge amount of code wrangling to move
these functions, but it means that adding a new function necessarily
involves modifying three files (it already involves include/petscyyy.h
and src/yyy/impls/zzz/zzz.c).

We still also need the monolithic mode to keep linking sane on systems
without shared libraries.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/6469bf76/attachment.sig>

From jed at jedbrown.org  Wed Jan 29 07:40:45 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 07:40:45 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E901FF.6030202@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
	<52E6D43F.9020501@mcs.anl.gov> <52E901FF.6030202@mcs.anl.gov>
Message-ID: <87k3dilxcy.fsf@jedbrown.org>

Karl Rupp <rupp at mcs.anl.gov> writes:
> Okay, I could reproduce this problem on my machine: A configuration 
> including
>   --with-threadcomm --with-pthreadclasses --with-openmp
> fails, while
>   --with-threadcomm --with-pthreadclasses
> works. Note that all runs are single-threaded, so it is only the 
> presence of -fopenmp which screws things up.

What about using CFLAGS=-fopenmp ?  Control flow is different in a few
places, so I would be concerned about that first.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/4c28bac2/attachment.sig>

From rupp at mcs.anl.gov  Wed Jan 29 08:07:34 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 29 Jan 2014 15:07:34 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <87k3dilxcy.fsf@jedbrown.org>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
	<52E6D43F.9020501@mcs.anl.gov> <52E901FF.6030202@mcs.anl.gov>
	<87k3dilxcy.fsf@jedbrown.org>
Message-ID: <52E90B26.3050806@mcs.anl.gov>

Hi,

 >> Okay, I could reproduce this problem on my machine: A configuration
>> including
>>    --with-threadcomm --with-pthreadclasses --with-openmp
>> fails, while
>>    --with-threadcomm --with-pthreadclasses
>> works. Note that all runs are single-threaded, so it is only the
>> presence of -fopenmp which screws things up.
>
> What about using CFLAGS=-fopenmp ?

What do you mean exactly? CFLAGS for Mani's application code?

>  Control flow is different in a few
> places, so I would be concerned about that first.

There is only a little bit of code in
  src/sys/threadcomm/impls/openmp/tcopenmp.c
  src/sys/threadcomm/impls/openmp/tcopenmpimpl.h
in addition to the registration of OpenMP functionality in
  src/sys/threadcomm/interface/threadcommregi.c
I really don't see how this could affect the execution of GPU code if 
the threadcomm-default 'nothread' is used.

Best regards,
Karli


From jed at jedbrown.org  Wed Jan 29 08:11:19 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 08:11:19 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E90B26.3050806@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
	<52E6D43F.9020501@mcs.anl.gov> <52E901FF.6030202@mcs.anl.gov>
	<87k3dilxcy.fsf@jedbrown.org> <52E90B26.3050806@mcs.anl.gov>
Message-ID: <87bnyulvy0.fsf@jedbrown.org>

Karl Rupp <rupp at mcs.anl.gov> writes:
> There is only a little bit of code in
>   src/sys/threadcomm/impls/openmp/tcopenmp.c
>   src/sys/threadcomm/impls/openmp/tcopenmpimpl.h
> in addition to the registration of OpenMP functionality in
>   src/sys/threadcomm/interface/threadcommregi.c
> I really don't see how this could affect the execution of GPU code if 
> the threadcomm-default 'nothread' is used.

There is a bunch of conditional compilation guarded by the
(nonconforming style) PETSC_THREADCOMM_ACTIVE.  Seems to me it's more
likely that something bad happens in one of those blocks than that
-fopenmp causes the compiler to misbehave.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/5858e827/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan 29 08:15:11 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 29 Jan 2014 08:15:11 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <87txcnkm84.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
	<CAHC8rGuid+4Bwbxf_Be6bEdvh9Rm_tC0ew1 LiFe1zmu vHbswXQ@mail.gmail.com>
	<87txcnkm84.fsf@jedbrown.org>
Message-ID: <9A250DD6-CE0D-42A0-AC29-90C6511154DB@mcs.anl.gov>


On Jan 29, 2014, at 6:26 AM, Jed Brown <jed at jedbrown.org> wrote:

> Geoff Oxberry <goxberry at gmail.com> writes:
>> I can answer some Homebrew questions. I typically skim petsc-dev and
>> petsc-users due to my relative PETSc inexperience, and only read a few
>> posts here and there. If someone bugs you about it and I miss it, ping me,
>> and I'll do my best to help. PETSc is also a package in the
>> homebrew-science repo, so I can lean on some of them for help also. I've
>> learned to build PETSc from source because the package builds in Linux
>> distros and Mac package managers usually lack the external package
>> capabilities I want (and they're old versions, too).
> 
> For systems with shared libraries, it may be useful to have a way to add
> plugins after building PETSc, rather than asking packagers to build
> every subset or a maximal subset.  The problem is that functions like
> PCHYPRESetType() are made visible to the user and thus become link-time
> dependencies.  Is there something *maintainable* we could do to allow
> applications to link without the packages?
> 
> Functions like the above either do PetscTryMethod or PetscUseMethod so
> they don't have a link-time dependency on the implementation
> PCHYPRESetType_HYPRE (and thus the external package), but are currently
> placed in the file with the external package.  If we moved them
> somewhere that was always compiled, we would get rid of the link-time
> dependency and then we could add these plugins after installing the
> PETSc base system.  It's not a huge amount of code wrangling to move
> these functions, but it means that adding a new function necessarily
> involves modifying three files (it already involves include/petscyyy.h
> and src/yyy/impls/zzz/zzz.c).

   It was always the intention since like 1996 to pull those out and make them separate. Possibly even as include files only. But aside from theoretical reasons we never had demand that actually made us do it. Could be a chicken and egg thing.

   Barry

> 
> We still also need the monolithic mode to keep linking sane on systems
> without shared libraries.



From rupp at mcs.anl.gov  Wed Jan 29 08:37:17 2014
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 29 Jan 2014 15:37:17 +0100
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <87bnyulvy0.fsf@jedbrown.org>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
	<52E6D43F.9020501@mcs.anl.gov> <52E901FF.6030202@mcs.anl.gov>
	<87k3dilxcy.fsf@jedbrown.org> <52E90B26.3050806@mcs.anl.gov>
	<87bnyulvy0.fsf@jedbrown.org>
Message-ID: <52E9121D.1030208@mcs.anl.gov>

Hey,

 >> There is only a little bit of code in
>>    src/sys/threadcomm/impls/openmp/tcopenmp.c
>>    src/sys/threadcomm/impls/openmp/tcopenmpimpl.h
>> in addition to the registration of OpenMP functionality in
>>    src/sys/threadcomm/interface/threadcommregi.c
>> I really don't see how this could affect the execution of GPU code if
>> the threadcomm-default 'nothread' is used.
>
> There is a bunch of conditional compilation guarded by the
> (nonconforming style) PETSC_THREADCOMM_ACTIVE.  Seems to me it's more
> likely that something bad happens in one of those blocks than that
> -fopenmp causes the compiler to misbehave.

I should have been more precise: If we configure PETSc to use OpenMP 
through threadcomm, then we run into issues. OpenMP and OpenCL code  per 
se, i.e. without any PETSc-code involved, is not causing troubles (and 
is a common build with ViennaCL). If I configure
  --with-openmp
but without
  --with-threadcomm --with-pthreadclasses
then execution is fine. Conversely, configuring
  --with-threadcomm --with-pthreadclasses
but without
  --with-openmp
also works. If all three options are supplied, wrong results are 
obtained. Weird. I'll keep it on my list, but will focus on other items 
first.

Best regards,
Karli




From jed at jedbrown.org  Wed Jan 29 08:45:45 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 08:45:45 -0600
Subject: [petsc-dev] Possible bugs when using TS with ViennaCL
In-Reply-To: <52E9121D.1030208@mcs.anl.gov>
References: <CAJzU6sCETx9U_7So7roMxLMuNEJ9sgccsL-BnS8k+tH_j15itw@mail.gmail.com>
	<52E37A65.9090106@mcs.anl.gov>
	<CAJzU6sApoZb_s9xhc3fG5hXFsgCDjafje8m3dQmiMBAvNp12HA@mail.gmail.com>
	<52E4293B.3080005@mcs.anl.gov>
	<CAJzU6sBJ_Vg_sjxV8amKjbGn7=4skVT6X+=e28381Qyy0y7o5Q@mail.gmail.com>
	<52E4E29B.9080102@mcs.anl.gov>
	<CAJzU6sBeu-3MYwTLOz5+AhFmCfNDNtRr_oKeGJW-50DkVxBsJQ@mail.gmail.com>
	<52E58EE9.1010001@mcs.anl.gov>
	<CAJzU6sD9Pj8+sbiBEQtZyGbspg7XLCpzH3MFfkUVZCvZrFY=Vg@mail.gmail.com>
	<52E6BC1C.7090701@mcs.anl.gov> <877g9lnnvb.fsf@jedbrown.org>
	<52E6D43F.9020501@mcs.anl.gov> <52E901FF.6030202@mcs.anl.gov>
	<87k3dilxcy.fsf@jedbrown.org> <52E90B26.3050806@mcs.anl.gov>
	<87bnyulvy0.fsf@jedbrown.org> <52E9121D.1030208@mcs.anl.gov>
Message-ID: <871tzqlucm.fsf@jedbrown.org>

Karl Rupp <rupp at mcs.anl.gov> writes:
> I should have been more precise: If we configure PETSc to use OpenMP 
> through threadcomm, then we run into issues. OpenMP and OpenCL code  per 
> se, i.e. without any PETSc-code involved, is not causing troubles (and 
> is a common build with ViennaCL). If I configure
>   --with-openmp
> but without
>   --with-threadcomm --with-pthreadclasses
> then execution is fine. Conversely, configuring
>   --with-threadcomm --with-pthreadclasses
> but without
>   --with-openmp
> also works. If all three options are supplied, wrong results are 
> obtained. Weird. I'll keep it on my list, but will focus on other items 
> first.

Okay, your diagnosis sounds right.  Gotta reduce this eventually so we
can file bug reports.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/9d17139c/attachment.sig>

From irving at naml.us  Wed Jan 29 12:00:51 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 29 Jan 2014 10:00:51 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
Message-ID: <CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>

On Tue, Jan 28, 2014 at 8:57 PM, Geoffrey Irving <irving at naml.us> wrote:
> On Tue, Jan 28, 2014 at 8:24 PM, Jed Brown <jed at jedbrown.org> wrote:
>> Matthew Knepley <knepley at gmail.com> writes:
>>
>>> On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>> > In ex62:
>>>> >
>>>> >   if (user.runType == RUN_FULL) {
>>>> >     PetscViewer viewer;
>>>> >     Vec         uLocal;
>>>> >     const char *name;
>>>> >
>>>> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
>>>> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
>>>> >     ierr = PetscViewerSetFormat(viewer,
>>>> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>>>> >     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);
>>>>
>>>> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
>>>> name to some *.vtu; the VTK viewer will infer output format from
>>>> extension and write the VTU XML format with binary-appended data in a
>>>> fairly fast and memory-scalable way.  Then
>>>>
>>>>   VecView(u,viewer);
>>>
>>> I think Jed is wrong here. You need to be using local vectors since
>>> they have BC and constraints in them. Please explain how your VTU
>>> format will magically put them in.
>>
>> We made this work when I threw a temper tantrum while we were writing TS
>> ex11.c (late 2012).
>>
>> PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
>> {
>>   DM             dm;
>>   PetscBool      isvtk;
>>   PetscErrorCode ierr;
>>
>>   PetscFunctionBegin;
>>   ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
>>   if (!dm) SETERRQ(PetscObjectComm((PetscObject)v), PETSC_ERR_ARG_WRONG, "Vector not generated from a DM");
>>   ierr = PetscObjectTypeCompare((PetscObject) viewer, PETSCVIEWERVTK, &isvtk);CHKERRQ(ierr);
>>   if (isvtk) {
>>     Vec         locv;
>>     const char *name;
>>
>>     ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
>>     ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
>>     ierr = PetscObjectSetName((PetscObject) locv, name);CHKERRQ(ierr);
>>     ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>>     ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>>     ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
>>     ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);
>
> Which of those lines adds the correct boundary condition values?

It looks like one of the following is true:

1. VecView on a DMPlex global vector does not insert correct boundary
conditions.
2. Scalar, residual, and jacobian integration are all performing an
extra boundary condition enforcement step.

I'm guessing it's (1).  If so, does that mean I have to do it Matt's
more verbose way?

Geoffrey


From jed at jedbrown.org  Wed Jan 29 13:36:12 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 12:36:12 -0700
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
Message-ID: <31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>

Matt's sample code doesn't set it either. We need to fix (and the *only* acceptable fix if that VecView always does the right thing, because we have to be able to call it in analysis settings that know nothing about your discretization).

The problem is that some vectors reside in a homogeneous space (e.g. increments and eigenvectors) while others reside in the inhomogeneous space (solutions). We can add a flag or BC attribute on the vector to this effect, but this (and slip conditions) was the issue that led me to conclude that removing boundary nodes was mostly a false economy.

On January 29, 2014 11:00:51 AM MST, Geoffrey Irving <irving at naml.us> wrote:
>On Tue, Jan 28, 2014 at 8:57 PM, Geoffrey Irving <irving at naml.us>
>wrote:
>> On Tue, Jan 28, 2014 at 8:24 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>
>>>> On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org>
>wrote:
>>>>
>>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>> > In ex62:
>>>>> >
>>>>> >   if (user.runType == RUN_FULL) {
>>>>> >     PetscViewer viewer;
>>>>> >     Vec         uLocal;
>>>>> >     const char *name;
>>>>> >
>>>>> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD,
>&viewer);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerSetType(viewer,
>PETSCVIEWERVTK);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerSetFormat(viewer,
>>>>> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerFileSetName(viewer,
>"ex62_sol.vtk");CHKERRQ(ierr);
>>>>>
>>>>> Please don't do this.  Just set the type to PETSCVIEWERVTK, set
>the file
>>>>> name to some *.vtu; the VTK viewer will infer output format from
>>>>> extension and write the VTU XML format with binary-appended data
>in a
>>>>> fairly fast and memory-scalable way.  Then
>>>>>
>>>>>   VecView(u,viewer);
>>>>
>>>> I think Jed is wrong here. You need to be using local vectors since
>>>> they have BC and constraints in them. Please explain how your VTU
>>>> format will magically put them in.
>>>
>>> We made this work when I threw a temper tantrum while we were
>writing TS
>>> ex11.c (late 2012).
>>>
>>> PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
>>> {
>>>   DM             dm;
>>>   PetscBool      isvtk;
>>>   PetscErrorCode ierr;
>>>
>>>   PetscFunctionBegin;
>>>   ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
>>>   if (!dm) SETERRQ(PetscObjectComm((PetscObject)v),
>PETSC_ERR_ARG_WRONG, "Vector not generated from a DM");
>>>   ierr = PetscObjectTypeCompare((PetscObject) viewer,
>PETSCVIEWERVTK, &isvtk);CHKERRQ(ierr);
>>>   if (isvtk) {
>>>     Vec         locv;
>>>     const char *name;
>>>
>>>     ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
>>>     ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
>>>     ierr = PetscObjectSetName((PetscObject) locv,
>name);CHKERRQ(ierr);
>>>     ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES,
>locv);CHKERRQ(ierr);
>>>     ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES,
>locv);CHKERRQ(ierr);
>>>     ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
>>>     ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);
>>
>> Which of those lines adds the correct boundary condition values?
>
>It looks like one of the following is true:
>
>1. VecView on a DMPlex global vector does not insert correct boundary
>conditions.
>2. Scalar, residual, and jacobian integration are all performing an
>extra boundary condition enforcement step.
>
>I'm guessing it's (1).  If so, does that mean I have to do it Matt's
>more verbose way?
>
>Geoffrey



From dharmareddy84 at gmail.com  Wed Jan 29 13:50:27 2014
From: dharmareddy84 at gmail.com (Dharmendar Reddy)
Date: Wed, 29 Jan 2014 13:50:27 -0600
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
Message-ID: <CAOwTPde2tMoUhc2qia11dSNBAbxhDtKwb7JZ6SHuZj-UWgPDZQ@mail.gmail.com>

In my code, I apply boundary conditions to the local vector before it
is used for assembling the residual and jacobian.
I use VecSetValuesSection with mode= INSERT_BC_VALUES

I also do the boundary condition's insert on the local vector before
it is used for vecview.

On Wed, Jan 29, 2014 at 12:00 PM, Geoffrey Irving <irving at naml.us> wrote:
> On Tue, Jan 28, 2014 at 8:57 PM, Geoffrey Irving <irving at naml.us> wrote:
>> On Tue, Jan 28, 2014 at 8:24 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>
>>>> On Tue, Jan 28, 2014 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>> > In ex62:
>>>>> >
>>>>> >   if (user.runType == RUN_FULL) {
>>>>> >     PetscViewer viewer;
>>>>> >     Vec         uLocal;
>>>>> >     const char *name;
>>>>> >
>>>>> >     ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerSetFormat(viewer,
>>>>> > PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr);
>>>>> >     ierr = PetscViewerFileSetName(viewer, "ex62_sol.vtk");CHKERRQ(ierr);
>>>>>
>>>>> Please don't do this.  Just set the type to PETSCVIEWERVTK, set the file
>>>>> name to some *.vtu; the VTK viewer will infer output format from
>>>>> extension and write the VTU XML format with binary-appended data in a
>>>>> fairly fast and memory-scalable way.  Then
>>>>>
>>>>>   VecView(u,viewer);
>>>>
>>>> I think Jed is wrong here. You need to be using local vectors since
>>>> they have BC and constraints in them. Please explain how your VTU
>>>> format will magically put them in.
>>>
>>> We made this work when I threw a temper tantrum while we were writing TS
>>> ex11.c (late 2012).
>>>
>>> PetscErrorCode VecView_Plex(Vec v, PetscViewer viewer)
>>> {
>>>   DM             dm;
>>>   PetscBool      isvtk;
>>>   PetscErrorCode ierr;
>>>
>>>   PetscFunctionBegin;
>>>   ierr = VecGetDM(v, &dm);CHKERRQ(ierr);
>>>   if (!dm) SETERRQ(PetscObjectComm((PetscObject)v), PETSC_ERR_ARG_WRONG, "Vector not generated from a DM");
>>>   ierr = PetscObjectTypeCompare((PetscObject) viewer, PETSCVIEWERVTK, &isvtk);CHKERRQ(ierr);
>>>   if (isvtk) {
>>>     Vec         locv;
>>>     const char *name;
>>>
>>>     ierr = DMGetLocalVector(dm, &locv);CHKERRQ(ierr);
>>>     ierr = PetscObjectGetName((PetscObject) v, &name);CHKERRQ(ierr);
>>>     ierr = PetscObjectSetName((PetscObject) locv, name);CHKERRQ(ierr);
>>>     ierr = DMGlobalToLocalBegin(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>>>     ierr = DMGlobalToLocalEnd(dm, v, INSERT_VALUES, locv);CHKERRQ(ierr);
>>>     ierr = VecView_Plex_Local(locv, viewer);CHKERRQ(ierr);
>>>     ierr = DMRestoreLocalVector(dm, &locv);CHKERRQ(ierr);
>>
>> Which of those lines adds the correct boundary condition values?
>
> It looks like one of the following is true:
>
> 1. VecView on a DMPlex global vector does not insert correct boundary
> conditions.
> 2. Scalar, residual, and jacobian integration are all performing an
> extra boundary condition enforcement step.
>
> I'm guessing it's (1).  If so, does that mean I have to do it Matt's
> more verbose way?
>
> Geoffrey


From irving at naml.us  Wed Jan 29 13:53:26 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 29 Jan 2014 11:53:26 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
	<31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>
Message-ID: <CAJ1ofpeQbSqG3-xeS6P4LHezgevXuw0XhSfLJEofCVGCM2Q_fw@mail.gmail.com>

On Wed, Jan 29, 2014 at 11:36 AM, Jed Brown <jed at jedbrown.org> wrote:
> Matt's sample code doesn't set it either. We need to fix (and the *only* acceptable fix if that VecView always does the right thing, because we have to be able to call it in analysis settings that know nothing about your discretization).

Matt's sample code doesn't set it either, but for Matt's sample code I
know where to insert the one line call to DMPlexProjectFunctionLocal.
For your version I never have explicit access to the local vector, so
I can't insert the fix.

> The problem is that some vectors reside in a homogeneous space (e.g. increments and eigenvectors) while others reside in the inhomogeneous space (solutions). We can add a flag or BC attribute on the vector to this effect, but this (and slip conditions) was the issue that led me to conclude that removing boundary nodes was mostly a false economy.

To leave the boundary conditions in, we would need efficient support
for a very large, very sparse MatNullSpace.  This is doable via
shells, but is it easy to do in a way that doesn't interfere with the
user's other null spaces?

Geoffrey


From jed at jedbrown.org  Wed Jan 29 14:00:48 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 13:00:48 -0700
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpeQbSqG3-xeS6P4LHezgevXuw0XhSfLJEofCVGCM2Q_fw@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
	<31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>
	<CAJ1ofpeQbSqG3-xeS6P4LHezgevXuw0XhSfLJEofCVGCM2Q_fw@mail.gmail.com>
Message-ID: <830c7cd4-d6f0-4b97-98a2-831a9faf8982@email.android.com>

Local vectors are supposed to be just that: local. VecViewing a local vector and expecting it to be parallel is perverse.  So we need a real interface.

Placing 1.0 on the diagonal (and don't assemble into those rows and columns) is the common way to deal with Dirichlet boundary nodes. See ex48 for one example. I have written about this in a few places; I can find the more complete description when I have a keyboard.

On January 29, 2014 12:53:26 PM MST, Geoffrey Irving <irving at naml.us> wrote:
>On Wed, Jan 29, 2014 at 11:36 AM, Jed Brown <jed at jedbrown.org> wrote:
>> Matt's sample code doesn't set it either. We need to fix (and the
>*only* acceptable fix if that VecView always does the right thing,
>because we have to be able to call it in analysis settings that know
>nothing about your discretization).
>
>Matt's sample code doesn't set it either, but for Matt's sample code I
>know where to insert the one line call to DMPlexProjectFunctionLocal.
>For your version I never have explicit access to the local vector, so
>I can't insert the fix.
>
>> The problem is that some vectors reside in a homogeneous space (e.g.
>increments and eigenvectors) while others reside in the inhomogeneous
>space (solutions). We can add a flag or BC attribute on the vector to
>this effect, but this (and slip conditions) was the issue that led me
>to conclude that removing boundary nodes was mostly a false economy.
>
>To leave the boundary conditions in, we would need efficient support
>for a very large, very sparse MatNullSpace.  This is doable via
>shells, but is it easy to do in a way that doesn't interfere with the
>user's other null spaces?
>
>Geoffrey



From irving at naml.us  Wed Jan 29 14:04:15 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 29 Jan 2014 12:04:15 -0800
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <830c7cd4-d6f0-4b97-98a2-831a9faf8982@email.android.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
	<31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>
	<CAJ1ofpeQbSqG3-xeS6P4LHezgevXuw0XhSfLJEofCVGCM2Q_fw@mail.gmail.com>
	<830c7cd4-d6f0-4b97-98a2-831a9faf8982@email.android.com>
Message-ID: <CAJ1ofpea7p_tamu3bmTbjHir0_gDt6Nm2smtpFDW=pduARGkhg@mail.gmail.com>

On Wed, Jan 29, 2014 at 12:00 PM, Jed Brown <jed at jedbrown.org> wrote:
> Local vectors are supposed to be just that: local. VecViewing a local vector and expecting it to be parallel is perverse.  So we need a real interface.
>
> Placing 1.0 on the diagonal (and don't assemble into those rows and columns) is the common way to deal with Dirichlet boundary nodes. See ex48 for one example. I have written about this in a few places; I can find the more complete description when I have a keyboard.

I'll look forward to the improved interface.  For better or worse, I'd
like to be able to view simulations in the near future, so it looks
like I have to go with Matt's perverse but working version now.

Geoffrey

> On January 29, 2014 12:53:26 PM MST, Geoffrey Irving <irving at naml.us> wrote:
>>On Wed, Jan 29, 2014 at 11:36 AM, Jed Brown <jed at jedbrown.org> wrote:
>>> Matt's sample code doesn't set it either. We need to fix (and the
>>*only* acceptable fix if that VecView always does the right thing,
>>because we have to be able to call it in analysis settings that know
>>nothing about your discretization).
>>
>>Matt's sample code doesn't set it either, but for Matt's sample code I
>>know where to insert the one line call to DMPlexProjectFunctionLocal.
>>For your version I never have explicit access to the local vector, so
>>I can't insert the fix.
>>
>>> The problem is that some vectors reside in a homogeneous space (e.g.
>>increments and eigenvectors) while others reside in the inhomogeneous
>>space (solutions). We can add a flag or BC attribute on the vector to
>>this effect, but this (and slip conditions) was the issue that led me
>>to conclude that removing boundary nodes was mostly a false economy.
>>
>>To leave the boundary conditions in, we would need efficient support
>>for a very large, very sparse MatNullSpace.  This is doable via
>>shells, but is it easy to do in a way that doesn't interfere with the
>>user's other null spaces?
>>
>>Geoffrey
>


From jed at jedbrown.org  Wed Jan 29 19:36:30 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 18:36:30 -0700
Subject: [petsc-dev] getting data out of a PetscFE simulation
In-Reply-To: <CAJ1ofpea7p_tamu3bmTbjHir0_gDt6Nm2smtpFDW=pduARGkhg@mail.gmail.com>
References: <CAJ1ofpdyvf2BT-XFVY5Hg0rS2sG1wtzUBeTjHuHEQLo+x5Wz-g@mail.gmail.com>
	<CAMYG4Gmox80RQNgwHPwK+WoaEjvY0_p0kjfmFJHgb2EQha1Ycw@mail.gmail.com>
	<CAJ1ofpfXXBwYnJj1eFnxCdDjfptgRa3DhhuqLpiH6C9K=X=J-A@mail.gmail.com>
	<CAMYG4GnAHhQh_zELHZmKh=wwORYjNBprHfgs4H=inqt6zc5Uzg@mail.gmail.com>
	<874n4nmp30.fsf@jedbrown.org>
	<CAMYG4Gn5uY_EPb1NdkZK_uDaxvdi5jRxVo2_-aYq0QKsoU2FnQ@mail.gmail.com>
	<871tzrmn3a.fsf@jedbrown.org>
	<CAJ1ofpeHFuToZHhra3_K66g7cfW+G3ONM6W5kwXOOBsnSQ2aXw@mail.gmail.com>
	<CAJ1ofpcb_DWhmLozNsUbxVkM3-PvF3KOQDA-44T3su=DZDA-+A@mail.gmail.com>
	<31dc856d-198c-41b7-b0cc-4bf6906984ef@email.android.com>
	<CAJ1ofpeQbSqG3-xeS6P4LHezgevXuw0XhSfLJEofCVGCM2Q_fw@mail.gmail.com>
	<830c7cd4-d6f0-4b97-98a2-831a9faf8982@email.android.com>
	<CAJ1ofpea7p_tamu3bmTbjHir0_gDt6Nm2smtpFDW=pduARGkhg@mail.gmail.com>
Message-ID: <87ob2ui735.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:

> On Wed, Jan 29, 2014 at 12:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>> Local vectors are supposed to be just that: local. VecViewing a local vector and expecting it to be parallel is perverse.  So we need a real interface.
>>
>> Placing 1.0 on the diagonal (and don't assemble into those rows and columns) is the common way to deal with Dirichlet boundary nodes. See ex48 for one example. I have written about this in a few places; I can find the more complete description when I have a keyboard.
>
> I'll look forward to the improved interface.  For better or worse, I'd
> like to be able to view simulations in the near future, so it looks
> like I have to go with Matt's perverse but working version now.

Okay.  Matt (and others), how would you like to signify when a Vec is a
member of a homogeneous versus inhomogeneous space?  When I did this in
Dohp, I interposed a global "closure" of the Vec and put the Dirichlet
value cache in there.  (That cache would be 0 for homogeneous vectors.)
As I mentioned, I'm now of the opinion that juggling these different Vec
representations to get more pure math isn't really worth it.

Instead, I use the approach described here.

http://scicomp.stackexchange.com/a/3300/119

This can handle messy boundary conditions like slip and phase change,
which don't make sense to remove in the way DMPlex currently operates.


If people want to keep enforcing boundary conditions by removing those
degrees of freedom, we need to add the concept of inhomogeneous and
homogeneous spaces.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/82992980/attachment.sig>

From jed at jedbrown.org  Wed Jan 29 19:55:01 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 18:55:01 -0700
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <9A250DD6-CE0D-42A0-AC29-90C6511154DB@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
	<87txcnkm84.fsf@jedbrown.org> <9A250D D6-CE0D-
	42A0-AC29-90C6511154DB@mcs.anl.gov>
Message-ID: <87k3dii68a.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    It was always the intention since like 1996 to pull those out and
>    make them separate. Possibly even as include files only. But aside
>    from theoretical reasons we never had demand that actually made us
>    do it. Could be a chicken and egg thing.

How would you like to structure them?  (It doesn't seem sane to me to
make them header-only.)

1. src/xxx/impls/yyy/yyydyn.c

   This doesn't jive with the current #requirespackage, but we could
   make that work.  It's also a lot more tiny files, which may slow
   compilation somewhat.  (OTOH, maybe there aren't that many dynamic
   specialized functions.)

2. src/xxx/impls/xxxdyn.c  (containing functions for all yyy)

   This seems a little out of place, but is clearly associated with
   extensions.

3. src/xxx/interface/xxxdyn.c

   The functions are going in the common interface header, so why not
   put their definitions in the directory with all the other definitions
   from that header?


I'm leaning toward 3, though other suggestions (and naming) welcome.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/4c40a59b/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan 29 20:10:46 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 29 Jan 2014 20:10:46 -0600
Subject: [petsc-dev] Handling plugins
In-Reply-To: <87k3dii68a.fsf@jedbrown.org>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
	<87txcnkm84.fsf@jedbrown.org>
	<9A250D D6-CE0D- 42A0-AC29-90C6511154DB@mcs.anl.gov>
	<87k3dii68a.fsf@jedbrown.org>
Message-ID: <41182DEF-A400-435E-813F-4178EC7CB183@mcs.anl.gov>


On Jan 29, 2014, at 7:55 PM, Jed Brown <jed at jedbrown.org> wrote:

> 
> 3. src/xxx/interface/xxxdyn.c
> 
>   The functions are going in the common interface header,

     This is only true sometimes. For example 

      petscdm*.h for different kinds of dms,  

      petscviewerhdf5.h      - this and the next one are because you need the underlying package to access some of the APIs

      petscviewersaws.h

     petscsnesfas.h           - this should probably be put in petscsnes.h  ???

      petscpcasa.h             - I vote we completely delete the ASA stuff, it ain?t used and must be largely broken  ???


> so why not
>   put their definitions in the directory with all the other definitions
>   from that header?

   It is kind of weird to have ?plugin? code directly in the same files as non-plugin code. You kind of think of plugin code as a chunk of something that you just drop in some place and then use. Having it just sitting in the same include file is strange.

  Barry

Request-assigned: Barry,Jed,Peter  merge petscsnesfas.h into petscsnes.h and remove PCASA?

> 
> 
> I'm leaning toward 3, though other suggestions (and naming) welcome.



From jed at jedbrown.org  Wed Jan 29 20:26:23 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 19:26:23 -0700
Subject: [petsc-dev] Handling plugins
In-Reply-To: <41182DEF-A400-435E-813F-4178EC7CB183@mcs.anl.gov>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<87k3dltmjk.fsf@jedbrown.org>
	<alpine.LFD.2.11.1401271034270.2458@asterix>
	<6E9EAC8E-5E8A-4489-99E2-3539E4CF300E@mcs.anl.gov>
	<m2wqhl2uc3.fsf@gmail.com>
	<CADOhEh4VNiYvJq+BPCCwmYtLdrx6aYU9vEBdiZEyCHtHJZq26g@mail.gmail.com>
	<CAPhiW4gEeVVEpe-EU_9saBew6XL4fHAPmhaJEhMqmMjwYHKzSg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<CAHC8rGuG3x1F-rVU4OQOOPvPvS3x1aLjFWibjMpNsym3FvWcaA@mail.gmail.com>
	<alpine.LFD.2.11.1401281818130.18968@asterix>
	<87txcnkm84.fsf@jedbrown.org> <87k3di i68a.fsf
	@jedbrown.org> <41182DEF-A400-435E-813F-4178EC7CB183@mcs.anl.gov>
Message-ID: <87bnyui4s0.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

> On Jan 29, 2014, at 7:55 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> 
>> 3. src/xxx/interface/xxxdyn.c
>> 
>>   The functions are going in the common interface header,
>
>      This is only true sometimes. For example 
>
>       petscdm*.h for different kinds of dms,  
>
>       petscviewerhdf5.h      - this and the next one are because you need the underlying package to access some of the APIs

In these cases, the external dependency leaks out into the interface so
we can't define the functions with HDF5.  The definitions of these
functions should stay where they are.  I would even make those headers
install with the plugin instead of with base PETSc.

>      petscsnesfas.h           - this should probably be put in petscsnes.h  ???

I don't care.  I guess it's consistent with folding petscpcmg.h into
petscpc.h.

>       petscpcasa.h             - I vote we completely delete the ASA stuff, it ain?t used and must be largely broken  ???

I think so.  It's a cool research algorithm, but I'm afraid it may be
more work to clean up the current implementation than to reimplement.

>> so why not
>>   put their definitions in the directory with all the other definitions
>>   from that header?
>
>    It is kind of weird to have ?plugin? code directly in the same
>    files as non-plugin code. You kind of think of plugin code as a
>    chunk of something that you just drop in some place and then
>    use. Having it just sitting in the same include file is strange.

Functions that don't have external dependencies aren't actually part of
the plugin.  They are richer interfaces that don't require the plugins
(though they may not do something _useful_ unless the plugin exists).
In particular, PackageA could ship a binary that calls PCHYPRESetType(),
which would have a hard dependency on libpetsc.so, but NOT on the Hypre
plugin.  Since it has more functionality when the petsc-hypre plugin is
around, package managers would say that it has an optional or
"recommended" dependency on the petsc-hypre plugin.  Users could choose
to install that later if they wanted.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/a6663dca/attachment.sig>

From bsmith at mcs.anl.gov  Wed Jan 29 21:29:50 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 29 Jan 2014 21:29:50 -0600
Subject: [petsc-dev] questions on Tao "integration" into PETSc repository
Message-ID: <9FCDB9B7-0991-4330-8246-58830CE60FDD@mcs.anl.gov>


   I?ve made a lot of progress in processing tao source to PETSc coding standards in the branch balay/tao-to-petsc 

   But now I have some observations/questions

   tao.h just includes petsc.h

    TaoInitialize/Finalize() basically calls PetscInitialize/Finalize

   The single abstract Tao solver object is called TaoSolve  and is defined in taosolver.h
          However all methods on the TaoSolver are prefixed with only Tao, for example TaoSetXXX()

   The single abstract Tao line search object is called TaoLineSearch  and is defined in taolinesearch.h 

   With everyone?s permission I would like to 

     1) remove TaoInitialize/Finalize()

     2) rename the TaoSolver object to Tao

     3) move the taosolver.h include to tao.h

   With this change Tao will be name-spaced and appear in the same character as SNES does

   Do I have everyone?s agreement to make these changes? Other suggestions?

   Thanks

   Barry



From irving at naml.us  Wed Jan 29 22:47:45 2014
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 29 Jan 2014 20:47:45 -0800
Subject: [petsc-dev] questions on Tao "integration" into PETSc repository
In-Reply-To: <9FCDB9B7-0991-4330-8246-58830CE60FDD@mcs.anl.gov>
References: <9FCDB9B7-0991-4330-8246-58830CE60FDD@mcs.anl.gov>
Message-ID: <CAJ1ofpfjxCa-GWN_7sm9-gzSoY_=GREzooNLb_GS7at4gmogvw@mail.gmail.com>

On Wed, Jan 29, 2014 at 7:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>    I've made a lot of progress in processing tao source to PETSc coding standards in the branch balay/tao-to-petsc
>
>    But now I have some observations/questions
>
>    tao.h just includes petsc.h
>
>     TaoInitialize/Finalize() basically calls PetscInitialize/Finalize
>
>    The single abstract Tao solver object is called TaoSolve  and is defined in taosolver.h
>           However all methods on the TaoSolver are prefixed with only Tao, for example TaoSetXXX()
>
>    The single abstract Tao line search object is called TaoLineSearch  and is defined in taolinesearch.h
>
>    With everyone's permission I would like to
>
>      1) remove TaoInitialize/Finalize()
>
>      2) rename the TaoSolver object to Tao
>
>      3) move the taosolver.h include to tao.h
>
>    With this change Tao will be name-spaced and appear in the same character as SNES does
>
>    Do I have everyone's agreement to make these changes? Other suggestions?
>
>    Thanks
>
>    Barry

This may be outside of the desired scope of changes, but would it be
possible to remove the two-level option structure imposed by Tao for
ksps and preconditioners?  E.g., currently to specify minres, one does

    -tao_nls_ksp_type petsc -ksp_type minres

Is there a reason the namespace can't be flattened?  Possibly more
unreasonably, is there a need for TaoLineSearch to be distinct from
SNESLineSearch?

Geoffrey


From jed at jedbrown.org  Wed Jan 29 23:42:09 2014
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 29 Jan 2014 22:42:09 -0700
Subject: [petsc-dev] questions on Tao "integration" into PETSc repository
In-Reply-To: <CAJ1ofpfjxCa-GWN_7sm9-gzSoY_=GREzooNLb_GS7at4gmogvw@mail.gmail.com>
References: <9FCDB9B7-0991-4330-8246-58830CE60FDD@mcs.anl.gov>
	<CAJ1ofpfjxCa-GWN_7sm9-gzSoY_=GREzooNLb_GS7at4gmogvw@mail.gmail.com>
Message-ID: <87r47qgh5a.fsf@jedbrown.org>

Geoffrey Irving <irving at naml.us> writes:
> This may be outside of the desired scope of changes, but would it be
> possible to remove the two-level option structure imposed by Tao for
> ksps and preconditioners?  E.g., currently to specify minres, one does
>
>     -tao_nls_ksp_type petsc -ksp_type minres
>
> Is there a reason the namespace can't be flattened?  

Agreed.

> Possibly more unreasonably, is there a need for TaoLineSearch to be
> distinct from SNESLineSearch?

They really should be, but I think we should merge Tao to 'master'
before digging into that.  Is Tao going to be part of the petsc-3.5
release?  How should we warn about interfaces that are likely to change?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140129/30e3bb8f/attachment.sig>

From baagaard at usgs.gov  Thu Jan 30 12:46:08 2014
From: baagaard at usgs.gov (Brad Aagaard)
Date: Thu, 30 Jan 2014 10:46:08 -0800
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <87wqhhfhe9.fsf@jedbrown.org>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
Message-ID: <52EA9DF0.1070705@usgs.gov>

On 01/30/2014 10:34 AM, Jed Brown wrote:
> Brad Aagaard <baagaard at usgs.gov> writes:
>
>> Matt and Jed,
>>
>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
>> deal with memory alignment and a zero size. It looks like the pointer
>> will NOT be NULL for a size of 0. Is this true?
>
> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
> need the size anyway to know how many elements are in the array.

I thought it was a nice feature that PETSc improved on malloc() and 
free() by returning NULL for zero sized allocation (although this wasn't 
true for --with-debugging=0 due to memory alignment) and set pointers to 
NULL after freeing.

What is the rationale for not returning NULL for mallocs of size zero 
other than conforming to C malloc behavior?

Brad




From knepley at gmail.com  Thu Jan 30 12:50:15 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 12:50:15 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <52EA9DF0.1070705@usgs.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
Message-ID: <CAMYG4Gkvup8S+qoO2PcTbyW=2srzJQe-5=efg70Dskg0NJertg@mail.gmail.com>

On Thu, Jan 30, 2014 at 12:46 PM, Brad Aagaard <baagaard at usgs.gov> wrote:

> On 01/30/2014 10:34 AM, Jed Brown wrote:
>
>> Brad Aagaard <baagaard at usgs.gov> writes:
>>
>>  Matt and Jed,
>>>
>>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
>>> deal with memory alignment and a zero size. It looks like the pointer
>>> will NOT be NULL for a size of 0. Is this true?
>>>
>>
>> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
>> need the size anyway to know how many elements are in the array.
>>
>
> I thought it was a nice feature that PETSc improved on malloc() and free()
> by returning NULL for zero sized allocation (although this wasn't true for
> --with-debugging=0 due to memory alignment) and set pointers to NULL after
> freeing.
>
> What is the rationale for not returning NULL for mallocs of size zero
> other than conforming to C malloc behavior?


There is no such thing as "conforming to C malloc", since that WOULD allow
returning NULL in this case.

I think we are doing users a disservice by not enforcing this.

   Matt


>
> Brad
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/230ef8b3/attachment.html>

From knepley at gmail.com  Thu Jan 30 13:01:06 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 13:01:06 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <CAMYG4Gkvup8S+qoO2PcTbyW=2srzJQe-5=efg70Dskg0NJertg@mail.gmail.com>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<CAMYG4Gkvup8S+qoO2PcTbyW=2srzJQe-5=efg70Dskg0NJertg@mail.gmail.com>
Message-ID: <CAMYG4G=9OkKyrnGP8hki3GMx29oqg65Jk3BgmRhb+KiG4RFyKA@mail.gmail.com>

On Thu, Jan 30, 2014 at 12:50 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 30, 2014 at 12:46 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
>
>> On 01/30/2014 10:34 AM, Jed Brown wrote:
>>
>>> Brad Aagaard <baagaard at usgs.gov> writes:
>>>
>>>  Matt and Jed,
>>>>
>>>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
>>>> deal with memory alignment and a zero size. It looks like the pointer
>>>> will NOT be NULL for a size of 0. Is this true?
>>>>
>>>
>>> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
>>> need the size anyway to know how many elements are in the array.
>>>
>>
>> I thought it was a nice feature that PETSc improved on malloc() and
>> free() by returning NULL for zero sized allocation (although this wasn't
>> true for --with-debugging=0 due to memory alignment) and set pointers to
>> NULL after freeing.
>>
>> What is the rationale for not returning NULL for mallocs of size zero
>> other than conforming to C malloc behavior?
>
>
> There is no such thing as "conforming to C malloc", since that WOULD allow
> returning NULL in this case.
>

Actually, we are violating the malloc standard that Jed sent out. It says
that for 0 size, malloc must return

  NULL or a unique pointer

However, if we have

  PetscMalloc2(0,&r1,1,&r2)

then r1 == r2 with the optimized implementation, and the pointers are not
unique.

   Matt


> I think we are doing users a disservice by not enforcing this.
>
>    Matt
>
>
>>
>> Brad
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/c2510e90/attachment.html>

From jed at jedbrown.org  Thu Jan 30 13:05:17 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 12:05:17 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <52EA9DF0.1070705@usgs.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
Message-ID: <87sis5ffyq.fsf@jedbrown.org>

Brad Aagaard <baagaard at usgs.gov> writes:
> I thought it was a nice feature that PETSc improved on malloc() and 
> free() by returning NULL for zero sized allocation (although this wasn't 
> true for --with-debugging=0 due to memory alignment) and set pointers to 
> NULL after freeing.
>
> What is the rationale for not returning NULL for mallocs of size zero 
> other than conforming to C malloc behavior?

The standard could have easily specified that malloc(0) returns NULL.
It didn't because leaving it unspecified allows better memory-checking
tools (e.g., you can check for matching free even when your tests happen
to allocate 0 bytes).  As you mention, PETSc did not guarantee it either
--with-debugging=0, so correct code cannot depend on it.  We are just
now being explicit that we only make the same guarantees as malloc().

We still set pointers to NULL after freeing.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/d4d1a9dd/attachment.sig>

From jed at jedbrown.org  Thu Jan 30 13:07:51 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 12:07:51 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <CAMYG4G=9OkKyrnGP8hki3GMx29oqg65Jk3BgmRhb+KiG4RFyKA@mail.gmail.com>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<CAMYG4Gkvup8S+qoO2PcTbyW=2srzJQe-5=efg70Dskg0NJertg@mail.gmail.com>
	<CAMYG4G=9OkKyrnGP8hki3GMx29oqg65Jk3BgmRhb+KiG4RFyKA@mail.gmail.com>
Message-ID: <87ppn9ffug.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> Actually, we are violating the malloc standard that Jed sent out. It says
> that for 0 size, malloc must return
>
>   NULL or a unique pointer
>
> However, if we have
>
>   PetscMalloc2(0,&r1,1,&r2)
>
> then r1 == r2 with the optimized implementation, and the pointers are not
> unique.

They are still correct to pass to PetscFree2(r1,r2).  I thought the
point of PetscMalloc[1-7] was to emulate doing a single big malloc and
calculating offsets, not to be semantically identical to calling malloc
several times (which would then allow separate calls to PetscFree).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/cd5eda08/attachment.sig>

From bsmith at mcs.anl.gov  Thu Jan 30 13:53:45 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 13:53:45 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <52EA9DF0.1070705@usgs.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
Message-ID: <3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>


On Jan 30, 2014, at 12:46 PM, Brad Aagaard <baagaard at usgs.gov> wrote:

> On 01/30/2014 10:34 AM, Jed Brown wrote:
>> Brad Aagaard <baagaard at usgs.gov> writes:
>> 
>>> Matt and Jed,
>>> 
>>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
>>> deal with memory alignment and a zero size. It looks like the pointer
>>> will NOT be NULL for a size of 0. Is this true?
>> 
>> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
>> need the size anyway to know how many elements are in the array.
> 
> I thought it was a nice feature that PETSc improved on malloc() and free() by returning NULL for zero sized allocation (although this wasn't true for --with-debugging=0 due to memory alignment) and set pointers to NULL after freeing.

   With PetscMalloc() we could certainly return NULL on 0 mallocs but the code is a bit more involved  for the PetscMallocn() case. 

   Here is what I suggestion. Someone suggest (i.e.. write) a refactorization of PetscMalloc(), PetscMallocn() that handles correctly any of the sizes being zero correctly in a branch and see how it goes. 


   Barry


> 
> What is the rationale for not returning NULL for mallocs of size zero other than conforming to C malloc behavior?
> 
> Brad
> 
> 



From knepley at gmail.com  Thu Jan 30 13:55:08 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 13:55:08 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
Message-ID: <CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>

On Thu, Jan 30, 2014 at 1:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Jan 30, 2014, at 12:46 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
>
> > On 01/30/2014 10:34 AM, Jed Brown wrote:
> >> Brad Aagaard <baagaard at usgs.gov> writes:
> >>
> >>> Matt and Jed,
> >>>
> >>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
> >>> deal with memory alignment and a zero size. It looks like the pointer
> >>> will NOT be NULL for a size of 0. Is this true?
> >>
> >> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
> >> need the size anyway to know how many elements are in the array.
> >
> > I thought it was a nice feature that PETSc improved on malloc() and
> free() by returning NULL for zero sized allocation (although this wasn't
> true for --with-debugging=0 due to memory alignment) and set pointers to
> NULL after freeing.
>
>    With PetscMalloc() we could certainly return NULL on 0 mallocs but the
> code is a bit more involved  for the PetscMallocn() case.
>
>    Here is what I suggestion. Someone suggest (i.e.. write) a
> refactorization of PetscMalloc(), PetscMallocn() that handles correctly any
> of the sizes being zero correctly in a branch and see how it goes.


I have pushed it. It looks simple to me

   Matt


>
>    Barry
>
>
> >
> > What is the rationale for not returning NULL for mallocs of size zero
> other than conforming to C malloc behavior?
> >
> > Brad
> >
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/e3e3c424/attachment.html>

From bsmith at mcs.anl.gov  Thu Jan 30 14:11:33 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 14:11:33 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
Message-ID: <DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>


On Jan 30, 2014, at 1:55 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 30, 2014 at 1:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Jan 30, 2014, at 12:46 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
> 
> > On 01/30/2014 10:34 AM, Jed Brown wrote:
> >> Brad Aagaard <baagaard at usgs.gov> writes:
> >>
> >>> Matt and Jed,
> >>>
> >>> I see that Jed pushed some changes (jed/malloc-zero) for PetscMalloc to
> >>> deal with memory alignment and a zero size. It looks like the pointer
> >>> will NOT be NULL for a size of 0. Is this true?
> >>
> >> Yes, just like malloc(), it can be either a unique pointer or NULL.  You
> >> need the size anyway to know how many elements are in the array.
> >
> > I thought it was a nice feature that PETSc improved on malloc() and free() by returning NULL for zero sized allocation (although this wasn't true for --with-debugging=0 due to memory alignment) and set pointers to NULL after freeing.
> 
>    With PetscMalloc() we could certainly return NULL on 0 mallocs but the code is a bit more involved  for the PetscMallocn() case.
> 
>    Here is what I suggestion. Someone suggest (i.e.. write) a refactorization of PetscMalloc(), PetscMallocn() that handles correctly any of the sizes being zero correctly in a branch and see how it goes.
> 
> I have pushed it. It looks simple to me

   I don?t see how this can possibly work

#if defined(PETSC_USE_DEBUG)
#define PetscFree2(m1,m2)   (PetscFree(m2) || PetscFree(m1))
#else
#define PetscFree2(m1,m2)   ((m2)=0, PetscFree(m1))
#endif

  You need to free the first m[i] that is not null. If m1 was null in the optimized form you never free anything.

   These macros are getting horrible, can they become something cleaner?


   Barry

> 
>    Matt
>  
> 
>    Barry
> 
> 
> >
> > What is the rationale for not returning NULL for mallocs of size zero other than conforming to C malloc behavior?
> >
> > Brad
> >
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From jed at jedbrown.org  Thu Jan 30 14:15:26 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 13:15:26 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
Message-ID: <87ha8lfcpt.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>>    Here is what I suggestion. Someone suggest (i.e.. write) a refactorization of PetscMalloc(), PetscMallocn() that handles correctly any of the sizes being zero correctly in a branch and see how it goes.
>> 
>> I have pushed it. It looks simple to me
>
>    I don?t see how this can possibly work
>
> #if defined(PETSC_USE_DEBUG)
> #define PetscFree2(m1,m2)   (PetscFree(m2) || PetscFree(m1))
> #else
> #define PetscFree2(m1,m2)   ((m2)=0, PetscFree(m1))
> #endif
>
>   You need to free the first m[i] that is not null. If m1 was null in the optimized form you never free anything.
>
>    These macros are getting horrible, can they become something cleaner?

Why do we want this behavior?  Can we see some concrete code that is
affected by this behavior?  We needed PetscMalloc[2-7] to work in
optimized mode, so nothing in PETSc should be depending on it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/7df5b481/attachment.sig>

From bsmith at mcs.anl.gov  Thu Jan 30 14:32:09 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 14:32:09 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <87ha8lfcpt.fsf@jedbrown.org>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
Message-ID: <194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>


On Jan 30, 2014, at 2:15 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>   Here is what I suggestion. Someone suggest (i.e.. write) a refactorization of PetscMalloc(), PetscMallocn() that handles correctly any of the sizes being zero correctly in a branch and see how it goes.
>>> 
>>> I have pushed it. It looks simple to me
>> 
>>   I don?t see how this can possibly work
>> 
>> #if defined(PETSC_USE_DEBUG)
>> #define PetscFree2(m1,m2)   (PetscFree(m2) || PetscFree(m1))
>> #else
>> #define PetscFree2(m1,m2)   ((m2)=0, PetscFree(m1))
>> #endif
>> 
>>  You need to free the first m[i] that is not null. If m1 was null in the optimized form you never free anything.
>> 
>>   These macros are getting horrible, can they become something cleaner?
> 
> Why do we want this behavior?  

   Matt feels that having zero space allocated items always as NULL is useful. Perhaps as a debugging/error determining issue since no one can ?accidentally? use the location returned by malloc(0) and in the debugger one won?t mistakenly think that a pointer which contains something returned by malloc(0) has a legitimate value in it.


> Can we see some concrete code that is
> affected by this behavior?  We needed PetscMalloc[2-7] to work in
> optimized mode, so nothing in PETSc should be depending on it.

   What does ?it? here refer to?  And what do you mean by ?depending on?.  Certainly we could make PetscMallocn() and PetscFreen() behave in a way Matt suggests even in optimized code (if we choose to).


   Some notes: 

   We currently mark freed pointers with a NULL so they cannot be accidentally used at a later time. 

   In Jed?s model something a ?zero length space? will have the value NULL and something not. There is, AFAIK, no tool to check the address and see if it is a real pointer or a pointer to an invalid location.

   With Matt?s scheme we could also mark ?zero length space? with a NULL. In an ideal world I could see having a different specific value for ?zero length space? such as INVALIDPOINTER then one could distinguish the two values and their different meanings.  Alias I don?t control the ideal C world. 

   I agree with Jed that we don?t want any PETSc code to be dependent on PetscMallocn(0) always being NULL in all cases; but I understand Matt?s argument that having PetscMallocn() always being NULL makes debugging easier.

   Barry




 




From jed at jedbrown.org  Thu Jan 30 15:00:22 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 14:00:22 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
	<194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
Message-ID: <87bnytfamx.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>> Why do we want this behavior?
>
>    Matt feels that having zero space allocated items always as NULL is
>    useful. Perhaps as a debugging/error determining issue since no one
>    can ?accidentally? use the location returned by malloc(0) and in
>    the debugger one won?t mistakenly think that a pointer which
>    contains something returned by malloc(0) has a legitimate value in
>    it.

The pointer returned by malloc(0) [e.g., optimized mode] is actually
invalid.  PETSc checks in debug mode; valgrind tracks zero-size
allocations.

>> Can we see some concrete code that is affected by this behavior?  We
>> needed PetscMalloc[2-7] to work in optimized mode, so nothing in
>> PETSc should be depending on it.
>
>    What does ?it? here refer to?  And what do you mean by ?depending
>    on?.  Certainly we could make PetscMallocn() and PetscFreen()
>    behave in a way Matt suggests even in optimized code (if we choose
>    to).

I'm assuming that current code is correct, thus I'm asking where
significant simplifications would occur if the caller knew that
zero-sized arrays were NULL.

>    Some notes:
>
>    We currently mark freed pointers with a NULL so they cannot be
>    accidentally used at a later time.
>
>    In Jed?s model something a ?zero length space? will have the value
>    NULL and something not. There is, AFAIK, no tool to check the
>    address and see if it is a real pointer or a pointer to an invalid
>    location.

There's no tool to determine whether it can hold more than one entry.
What arrays do we have that (a) do not store size and (b) do not use a
mandatory end marker like '\0' (in which case you had to allocate the
end marker)?

>    With Matt?s scheme we could also mark ?zero length space? with a
>    NULL. In an ideal world I could see having a different specific
>    value for ?zero length space? such as INVALIDPOINTER then one could
>    distinguish the two values and their different meanings.  Alias I
>    don?t control the ideal C world.

Why don't you have this information out-of-band anyway?

>    I agree with Jed that we don?t want any PETSc code to be dependent
>    on PetscMallocn(0) always being NULL in all cases; but I understand
>    Matt?s argument that having PetscMallocn() always being NULL makes
>    debugging easier.

I'm not convinced of this.  Memory checkers will notice accesses (and be
able to track back to where the zero-length array was "allocated").
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/7f0b8962/attachment.sig>

From knepley at gmail.com  Thu Jan 30 16:31:58 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 16:31:58 -0600
Subject: [petsc-dev] What happened to -log_summary_python?
Message-ID: <CAMYG4GkptF4X0j_BF+_bm2HdQ6ROBgr4d3xtPZ=yK+aAKG=ixg@mail.gmail.com>

I was using that.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/8d9cd2b2/attachment.html>

From bsmith at mcs.anl.gov  Thu Jan 30 16:45:43 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 16:45:43 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <87bnytfamx.fsf@jedbrown.org>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
	<194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
	<87bnytfamx.fsf@jedbrown.org>
Message-ID: <ED1BE87E-822D-421A-9CC3-5F14BD14F9C3@mcs.anl.gov>


On Jan 30, 2014, at 3:00 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>> Why do we want this behavior?
>> 
>>   Matt feels that having zero space allocated items always as NULL is
>>   useful. Perhaps as a debugging/error determining issue since no one
>>   can ?accidentally? use the location returned by malloc(0) and in
>>   the debugger one won?t mistakenly think that a pointer which
>>   contains something returned by malloc(0) has a legitimate value in
>>   it.
> 
> The pointer returned by malloc(0) [e.g., optimized mode] is actually
> invalid.  

    Yes, but in the debugger (or with a printf %p there is no portable way for a user to know that the pointer is invalid) whereas with NULL they would immediately know. A null pointer that leads to a segv conveys something different to someone debugging than some address where the user doesn?t know if that address came from a malloc(0) or some completely different way (like crazy pointer arithmetic).

> PETSc checks in debug mode; valgrind tracks zero-size
> allocations.
> 
>>> Can we see some concrete code that is affected by this behavior?  We
>>> needed PetscMalloc[2-7] to work in optimized mode, so nothing in
>>> PETSc should be depending on it.
>> 
>>   What does ?it? here refer to?  And what do you mean by ?depending
>>   on?.  Certainly we could make PetscMallocn() and PetscFreen()
>>   behave in a way Matt suggests even in optimized code (if we choose
>>   to).
> 
> I'm assuming that current code is correct, thus I'm asking where
> significant simplifications would occur if the caller knew that
> zero-sized arrays were NULL.

   There would no know simplifications in the code; it would allegedly make debugging sometimes easier

> 
>>   Some notes:
>> 
>>   We currently mark freed pointers with a NULL so they cannot be
>>   accidentally used at a later time.
>> 
>>   In Jed?s model something a ?zero length space? will have the value
>>   NULL and something not. There is, AFAIK, no tool to check the
>>   address and see if it is a real pointer or a pointer to an invalid
>>   location.
> 
> There's no tool to determine whether it can hold more than one entry.
> What arrays do we have that (a) do not store size and (b) do not use a
> mandatory end marker like '\0' (in which case you had to allocate the
> end marker)?
> 
>>   With Matt?s scheme we could also mark ?zero length space? with a
>>   NULL. In an ideal world I could see having a different specific
>>   value for ?zero length space? such as INVALIDPOINTER then one could
>>   distinguish the two values and their different meanings.  Alias I
>>   don?t control the ideal C world.
> 
> Why don't you have this information out-of-band anyway?

   If you are trying to debug some code that you didn?t personally write (that may have been written by an idiot) it is rarely possible for you to know where this out-of-bound information is.

> 
>>   I agree with Jed that we don?t want any PETSc code to be dependent
>>   on PetscMallocn(0) always being NULL in all cases; but I understand
>>   Matt?s argument that having PetscMallocn() always being NULL makes
>>   debugging easier.
> 
> I'm not convinced of this.  Memory checkers will notice accesses (and be
> able to track back to where the zero-length array was "allocated?).

   We would obviously turn off all of this completely when running under valgrind to let valgrind do its thing.

    Side note; PetscMallocn() needs much more error checking: for example PetscMalloc2(-8,&x1,10,&x2) would not be detected as a problem but is problematic. Since we have to add all this error checking (likely by making these beasts functions) adding a return NULL on 0 won?t muck up the code much.

  Barry





From jed at jedbrown.org  Thu Jan 30 18:02:05 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 17:02:05 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <ED1BE87E-822D-421A-9CC3-5F14BD14F9C3@mcs.anl.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
	<194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
	<87bnytfamx.fsf@jedbrown.org>
	<ED1BE87E-822D-421A-9CC3-5F14BD14F9C3@mcs.anl.gov>
Message-ID: <87txcldnnm.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

> On Jan 30, 2014, at 3:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>     Yes, but in the debugger (or with a printf %p there is no portable
>     way for a user to know that the pointer is invalid) whereas with
>     NULL they would immediately know. A null pointer that leads to a
>     segv conveys something different to someone debugging than some
>     address where the user doesn?t know if that address came from a
>     malloc(0) or some completely different way (like crazy pointer
>     arithmetic).

True, but an array length is needed anyway and any references to that
pointer can still be detected by the memory checker.  Also, if someone
really wants to get NULL (or a special value), we can use a malloc
implementation that does so.  (We can add a runtime option for this if
someone wants it for debugging.)

>    If you are trying to debug some code that you didn?t personally
>    write (that may have been written by an idiot) it is rarely
>    possible for you to know where this out-of-bound information is.

Fair, but you'll have as much trouble doing anything with that code.

>    We would obviously turn off all of this completely when running
>    under valgrind to let valgrind do its thing.
>
>     Side note; PetscMallocn() needs much more error checking: for
>     example PetscMalloc2(-8,&x1,10,&x2) would not be detected as a
>     problem but is problematic. 

Is this a problem actually encountered in practice?

>     Since we have to add all this error checking (likely by making
>     these beasts functions) adding a return NULL on 0 won?t muck up
>     the code much.

We can't make them functions because we use the type sizes and pass
__FILE__, __LINE__.  I'm not motivated to change this unless someone
demonstrates that this is a non-theoretical problem.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/68d53fd4/attachment.sig>

From bsmith at mcs.anl.gov  Thu Jan 30 18:50:02 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 18:50:02 -0600
Subject: [petsc-dev] What happened to -log_summary_python?
In-Reply-To: <CAMYG4GkptF4X0j_BF+_bm2HdQ6ROBgr4d3xtPZ=yK+aAKG=ixg@mail.gmail.com>
References: <CAMYG4GkptF4X0j_BF+_bm2HdQ6ROBgr4d3xtPZ=yK+aAKG=ixg@mail.gmail.com>
Message-ID: <0EEB4C77-CCE2-463E-B021-C6C92620AE70@mcs.anl.gov>

 
   Matt,

    Ok, I took a look at that and see why I deleted it. 

   I had hoped my new viewer with PetscLogView_Detailed() which can be accessed with -log_view :filename:ascii_info_detail could be used instead of PetscLogViewPython()

   It, in theory, provides more information since it provides the values for each process so you can do any kind of cross process statistics you want. Or you can just add up the values from each process. 

   Anyways, if you can?t use it let me know and we can iterate on PetscLogView_Detailed() to make it useful.

   Barry


On Jan 30, 2014, at 4:31 PM, Matthew Knepley <knepley at gmail.com> wrote:

> I was using that.
> 
>    Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener



From bsmith at mcs.anl.gov  Thu Jan 30 18:55:24 2014
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 30 Jan 2014 18:55:24 -0600
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <87txcldnnm.fsf@jedbrown.org>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
	<194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
	<87bnytfamx.fsf@jedbrown.org>
	<ED1BE87E-822D-421A-9CC3-5F14BD14F9C3@mcs.anl.gov>
	<87txcldnnm.fsf@jedbrown.org>
Message-ID: <377519B3-6956-495A-948B-E11ABA29C56F@mcs.anl.gov>


On Jan 30, 2014, at 6:02 PM, Jed Brown <jed at jedbrown.org> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>> On Jan 30, 2014, at 3:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>>    Yes, but in the debugger (or with a printf %p there is no portable
>>    way for a user to know that the pointer is invalid) whereas with
>>    NULL they would immediately know. A null pointer that leads to a
>>    segv conveys something different to someone debugging than some
>>    address where the user doesn?t know if that address came from a
>>    malloc(0) or some completely different way (like crazy pointer
>>    arithmetic).
> 
> True, but an array length is needed anyway and any references to that
> pointer can still be detected by the memory checker.  Also, if someone
> really wants to get NULL (or a special value), we can use a malloc
> implementation that does so.  (We can add a runtime option for this if
> someone wants it for debugging.)
> 
>>   If you are trying to debug some code that you didn?t personally
>>   write (that may have been written by an idiot) it is rarely
>>   possible for you to know where this out-of-bound information is.
> 
> Fair, but you'll have as much trouble doing anything with that code.
> 
>>   We would obviously turn off all of this completely when running
>>   under valgrind to let valgrind do its thing.
>> 
>>    Side note; PetscMallocn() needs much more error checking: for
>>    example PetscMalloc2(-8,&x1,10,&x2) would not be detected as a
>>    problem but is problematic. 
> 
> Is this a problem actually encountered in practice?
> 
>>    Since we have to add all this error checking (likely by making
>>    these beasts functions) adding a return NULL on 0 won?t muck up
>>    the code much.
> 
> We can't make them functions because we use the type sizes and pass
> __FILE__, __LINE__.  I'm not motivated to change this unless someone
> demonstrates that this is a non-theoretical problem.

  No one is asking YOU to change this. I would be happy if it was reworked using function calls (obviously with a macro wrapper) that made all the decisions (debug or not etc) runtime rather than compile time and had much more robust error checking, then adding a NULL on 0 feature would be a trivial addition.

  Note that currently when running optimized with valgrind; valgrind only sees the one_big_malloc so can miss lots of overwrite and underwrites within the chunks malloced. By making everything runtime we can let valgrind have more information and manage correctness of each sub chunk.


  Barry






From mfadams at lbl.gov  Thu Jan 30 19:22:45 2014
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 30 Jan 2014 20:22:45 -0500
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <m261p338ys.fsf@gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix>
	<m261p338ys.fsf@gmail.com>
Message-ID: <CADOhEh7-WNmHSuBKMG+rqQdF46=eFb8ShycsNp=hGGRvLZTRDA@mail.gmail.com>

>
>
> $ sudo port install mpich-default
> $ sudo port select --set mpi mpich-mp-fortran
>
> Then try to build PETSc (minus the --download-mpich and CC, CXX, FC
> options).


That worked


> You could also try to install most of the external packages:
>
> $ sudo port install hypre ml parmetis metis hdf5-18 netcdf triangle \
> superlu superlu_dist mumps
>

This instal seemed to work but how would I configure PETSc to use them?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/fcdc15c6/attachment.html>

From sean.michael.farley at gmail.com  Thu Jan 30 22:13:39 2014
From: sean.michael.farley at gmail.com (Sean Farley)
Date: Thu, 30 Jan 2014 22:13:39 -0600
Subject: [petsc-dev] configure failed after update of OSX
In-Reply-To: <CADOhEh7-WNmHSuBKMG+rqQdF46=eFb8ShycsNp=hGGRvLZTRDA@mail.gmail.com>
References: <CADOhEh4LP5A7=OStp-Yj0xWJO7fss6wEeSjB1BHg63XCepGUHg@mail.gmail.com>
	<alpine.LFD.2.11.1401272312530.2447@asterix>
	<CAPhiW4i0wkFQgaN=VOfTugsJvwc-cdfoWDmEKNHhxRRvZwrn_w@mail.gmail.com>
	<alpine.LFD.2.11.1401272340000.2447@asterix>
	<alpine.LFD.2.11.1401272344400.2447@asterix>
	<CAHC8rGuv0G2u875U=HvqQtXwDaqndGcegGnJUy1G+XHC8NT3Hg@mail.gmail.com>
	<m2lhy02fsk.fsf@gmail.com>
	<CAHC8rGtbydvsdYRyRap8zJa9csx6iT7xhJ=qMD2VgSfpECKrLA@mail.gmail.com>
	<B4FD4033-A167-42F4-AFA8-DB2D9C61C88E@mcs.anl.gov>
	<alpine.LFD.2.11.1401281222060.18968@asterix>
	<87ppnclz2e.fsf@jedbrown.org>
	<CAMYG4Gk92hLjm69MtYVbFBEyVXnHkpgLD_ZW2wQyGXiL5ac1nQ@mail.gmail.com>
	<CADOhEh6egm1G+1m7hjMwZzb04cJYN+8sxr=SeBwjJG3PTFt5Rg@mail.gmail.com>
	<alpine.LFD.2.11.1401281324440.18968@asterix>
	<CADOhEh4OPTVSB1dytCGLBx9mzwPWWk=YuP=20XeOYHBWEQS0oQ@mail.gmail.com>
	<m2fvo73m3k.fsf@gmail.com>
	<alpine.LFD.2.11.1401281417480.18968@asterix>
	<m2a9ef3ckh.fsf@gmail.com>
	<alpine.LFD.2.11.1401281805220.18968@asterix> <m261p338ys.fs
	f@gmail.com>
	<CADOhEh7-WNmHSuBKMG+rqQdF46=eFb8ShycsNp=hGGRvLZTRDA@mail.gmail.com>
Message-ID: <m27g9g23gs.fsf@gmail.com>


mfadams at lbl.gov writes:

>>
>>
>> $ sudo port install mpich-default
>> $ sudo port select --set mpi mpich-mp-fortran
>>
>> Then try to build PETSc (minus the --download-mpich and CC, CXX, FC
>> options).
>
>
> That worked

Awesome

>> You could also try to install most of the external packages:
>>
>> $ sudo port install hypre ml parmetis metis hdf5-18 netcdf triangle \
>> superlu superlu_dist mumps
>>
>
> This instal seemed to work but how would I configure PETSc to use them?

I have a bash function to help me configure but it's basically this:

$ ./configure --with-{hdf5,netcdf,ml,hypre,metis,parmetis,scalapack,mumps,superlu,superlu_dist}-dir=/opt/local


From knepley at gmail.com  Thu Jan 30 23:05:41 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 23:05:41 -0600
Subject: [petsc-dev] What happened to -log_summary_python?
In-Reply-To: <0EEB4C77-CCE2-463E-B021-C6C92620AE70@mcs.anl.gov>
References: <CAMYG4GkptF4X0j_BF+_bm2HdQ6ROBgr4d3xtPZ=yK+aAKG=ixg@mail.gmail.com>
	<0EEB4C77-CCE2-463E-B021-C6C92620AE70@mcs.anl.gov>
Message-ID: <CAMYG4Gk42y_u=pOMzxKjO94jwVW9Sm4zcL8Db_zKsnhC+Kg2=Q@mail.gmail.com>

On Thu, Jan 30, 2014 at 6:50 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Matt,
>
>     Ok, I took a look at that and see why I deleted it.
>
>    I had hoped my new viewer with PetscLogView_Detailed() which can be
> accessed with -log_view :filename:ascii_info_detail could be used instead
> of PetscLogViewPython()
>

I like your new organization. I was trying to output in a format that did
not require us to write another parser. I will
reintegrate the code into your new setup.

  Thanks,

    Matt


>    It, in theory, provides more information since it provides the values
> for each process so you can do any kind of cross process statistics you
> want. Or you can just add up the values from each process.
>
>    Anyways, if you can't use it let me know and we can iterate on
> PetscLogView_Detailed() to make it useful.
>
>    Barry
>
>
> On Jan 30, 2014, at 4:31 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> > I was using that.
> >
> >    Matt
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/f854f93e/attachment.html>

From knepley at gmail.com  Thu Jan 30 23:10:23 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 23:10:23 -0600
Subject: [petsc-dev] loop structure in PetscFEIntegrateJacobian_Basic
In-Reply-To: <CAMYG4G=BKadvA4g_S+DLuExbkocE7=aqUf81QnL1zsy7OomQRA@mail.gmail.com>
References: <CAJ1ofpfMrze0VbP87eDvUWJ0EzZPCdgksYFjZZohtiiE5J-tow@mail.gmail.com>
	<CAMYG4G=BKadvA4g_S+DLuExbkocE7=aqUf81QnL1zsy7OomQRA@mail.gmail.com>
Message-ID: <CAMYG4G=XJLGC1gsTObyb5W5+Bpg=SVpmkrW48DVuGTBas+O4gA@mail.gmail.com>

On Tue, Jan 28, 2014 at 3:45 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Jan 28, 2014 at 3:31 PM, Geoffrey Irving <irving at naml.us> wrote:
>
>> PetscFEIntegrateJacobian_Basic has four layers of loops:
>>
>>    e - element
>>    f - component of field being differentiated
>>    g - component of field we're differentiating against
>>    q - quadrature point
>>
>> It looks like g3_func and friends are being called once for each
>> combination of (e,f,g,q), even though the calls to g3_func are
>> independent of (f,g).  If g3_func is expensive, this duplication seems
>> a bit slow.
>>
>> It seems like the loops should be flipped to be (e,q,f,g) major order,
>> with the user functions hoisted up to the (e,q) level.  Is there a
>> reason not to do this?
>
>
> Yes, it is stupid. I will fix it.
>

I did not forget. Pushed today and verified with the SNES ex62 tests.

  Matt


>    Matt
>
>
>>
>> Geoffrey
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/4af48c42/attachment.html>

From knepley at gmail.com  Thu Jan 30 23:11:49 2014
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 30 Jan 2014 23:11:49 -0600
Subject: [petsc-dev] Cell overlap in DMPlexDistribute
In-Reply-To: <52E7C567.7050101@imperial.ac.uk>
References: <52E7C567.7050101@imperial.ac.uk>
Message-ID: <CAMYG4GkfpViWZ2=+b4Dr7hnNTDrCrjjYPm5aHQJSL1g8np9Lcw@mail.gmail.com>

On Tue, Jan 28, 2014 at 8:57 AM, Michael Lange <michael.lange at imperial.ac.uk
> wrote:

> Hi,
>
> I noticed that the cell overlap created during DMPlexDistribute does not
> include cells that only share a vertex but no edge with an owned cell. This
> causes problems when performing local assembly
> (MAT_IGNORE_OFF_PROC_ENTRIES) based on information from the plex, because
> one contribution to the shared vertex is missing.
>
> As an example consider the 2x2 square with 8 cells (attached). When
> partitioned across two ranks with overlap=1 each rank owns 4 cells and in
> the current version knows about 2 halo cells, giving a total of 6 cells.
> The central vertex, however, touches 3 owned and 3 non-owned cells, one of
> which doesn't share an edge with any owned cells. So, in order to correctly
> assemble the central vertex locally, the rank owning it needs to know about
> 7 cells in total.
>
> I have attached a patch that fixes this problem by going over the inverse
> closure of all vertices associated with a given cell instead of using the
> provided edge graph. Please tell me what you think and whether there might
> be an easier way to fix this.
>

This is true, but unfortunately not what everyone wants. FVM people want
just what is there now. This
same choice comes up in preallocation. There I have put in the "center
dimension" which says what
kind of point do you use for the center, vertex or face? I guess we need
that here as well.

   Matt


> Kind regards
> Michael Lange
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/f725a69c/attachment.html>

From jed at jedbrown.org  Fri Jan 31 00:36:32 2014
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 30 Jan 2014 23:36:32 -0700
Subject: [petsc-dev] PetscMalloc for size zero
In-Reply-To: <377519B3-6956-495A-948B-E11ABA29C56F@mcs.anl.gov>
References: <52EA91F0.8080708@usgs.gov> <87wqhhfhe9.fsf@jedbrown.org>
	<52EA9DF0.1070705@usgs.gov>
	<3C2E4970-43A3-4BCB-8A23-5FF5CECDBEB8@mcs.anl.gov>
	<CAMYG4GkO41AA28kAVriLtqukmxnq5JA8gT3OxfT6YJd02mRt=A@mail.gmail.com>
	<DBAE47C0-9311-4449-BDFA-ACCF41D3D627@mcs.anl.gov>
	<87ha8lfcpt.fsf@jedbrown.org>
	<194F5D21-9EFB-465A-9A0C-E254ED494E46@mcs.anl.gov>
	<87bnytfamx.fsf@jedbrown.org>
	<ED1BE87E-822D-421A-9CC3-5F14BD14F9C3@mcs.anl.gov>
	<87txcldnnm.fsf@jedbrown.org>
	<377519B3-6956-495A-948B-E11ABA29C56F@mcs.anl.gov>
Message-ID: <87lhxwejyn.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:
>   Note that currently when running optimized with valgrind; valgrind
>   only sees the one_big_malloc so can miss lots of overwrite and
>   underwrites within the chunks malloced. By making everything runtime
>   we can let valgrind have more information and manage correctness of
>   each sub chunk.

I agree that it's good to defer as much as possible to runtime.  I
sorely wish I knew of a good way to implement a -debug option (short of
LD_PRELOADing the debug library).

I'm not happy about Matt's increasingly complicated macros, but I'm not
going to fight it any more.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140130/98282562/attachment.sig>