Hi, <div> I want to solve a linear eq. Kx=b where K is changed at each time step and the preconditioner is a shell function and doesn't change. After I change K, do I need to explicitly call KSP/PCSetOperator again to reset the matrix inside the KSP solver? </div>
<div>Does KSP hold copies of the matrices or just pointers to the matrices? </div><div>i.e., </div><div><br></div><div>PCSetOperator(pc,K,K,SAME_PRECONDITIONER);</div><div> ..............///other setup</div><div>KSPSolve(..);</div>
<div>...........////change K;</div><div>/////////Is PCSetOperator(......) needed here?</div><div><br></div><div>And what about if I want to change the preconditioner? do I need to destroy the PC/KSP object and re-create and setup everything? or KSP/PCSetOperator will release the memory space associated with the old matrices(e.g., ILU factors)? Thanks. </div>
<div><br></div><div> </div><div><br><div class="gmail_quote">On Wed, Mar 16, 2011 at 12:00 PM, <span dir="ltr"><<a href="mailto:petsc-users-request@mcs.anl.gov">petsc-users-request@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Send petsc-users mailing list submissions to<br>
<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/petsc-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:petsc-users-request@mcs.anl.gov">petsc-users-request@mcs.anl.gov</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:petsc-users-owner@mcs.anl.gov">petsc-users-owner@mcs.anl.gov</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of petsc-users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Building with MKL 10.3 (Preston, Eric - IS)<br>
2. Re: Building with MKL 10.3 (Robert Ellis)<br>
3. Re: Building with MKL 10.3 (Jed Brown)<br>
4. Re: Building with MKL 10.3 (Natarajan CS)<br>
5. Re: Building with MKL 10.3 (Rob Ellis)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 15 Mar 2011 16:55:25 -0400<br>
From: "Preston, Eric - IS" <<a href="mailto:Eric.Preston@itt.com">Eric.Preston@itt.com</a>><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
To: "<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
Message-ID:<br>
<<a href="mailto:A0853E30B604B04CABC1DDDF34DD586F0984FF9FC2@USFWA1EXMBX5.itt.net">A0853E30B604B04CABC1DDDF34DD586F0984FF9FC2@USFWA1EXMBX5.itt.net</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
<br>
As the original poster, I can say I didn't give much thought to using the threaded vs. non-threaded MKL libraries, I was just sharing a solution to building with a different MKL variant (which should probably be incorporated into the config scripts.) However, the discussion it spawned was informative. The main consideration there was not performance but compatibility with your other libraries. I know on windows, for instance, different run-time libraries are used in each case and all your components must be compiled with the same option. Thankfully, I'm not using windows for this project, so it might not make any difference.<br>
<br>
Eric Preston<br>
<br>
<br>
> Hello,<br>
> I am neither a regular Petsc user nor contributor so preemptive<br>
>apologies if I am completely off the line here.<br>
><br>
>I am not sure if the original poster had hyper-threading in mind when he<br>
>asked about multi-threading, in case that was the idea, I don't think using<br>
>petsc with MKL (HT) is going to give any benefit, I don't think MKL is<br>
>really resource insensitive.<br>
><br>
>Also I wonder what percentage of the code is actually blas/lapack intensive<br>
>to make any significant dent in wall cock?<br>
><br>
>of course +1 to everything else posed above.<br>
><br>
>Cheers,<br>
><br>
>C.S.N<br>
<br>
<br>
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.<br>
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 15 Mar 2011 21:30:41 +0000<br>
From: Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
To: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
Message-ID:<br>
<<a href="mailto:18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com">18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
All,<br>
<br>
Coincidentally, I have just spent much of the last two weeks testing the effect of the latest MKL blas-lapack release on a large domain decomposition style PETSc application, MPICH2, Windows, Intel Fortran, VS 2010 C++, all static linked, no hyperthreading, multi-blade, latest dual hex core Xeons, 25GB ram. Each core contains a domain and a shell is used for the matrix operations. Regardless of setting the number of threads for MKL or OMP, the MKL performance was worse than simply using --download-f-blas-lapack=1. My interpretation is that the decomposition of one domain per core effectively saturates the hardware and performance is actually degraded using the more sophisticated MKL.<br>
<br>
And, yes, I know that Windows is less than ideal for this type of work but there are other constraints...<br>
<br>
Cheers,<br>
Rob<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a>] On Behalf Of Preston, Eric - IS<br>
Sent: Tuesday, March 15, 2011 1:55 PM<br>
To: <a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
<br>
<br>
As the original poster, I can say I didn't give much thought to using the threaded vs. non-threaded MKL libraries, I was just sharing a solution to building with a different MKL variant (which should probably be incorporated into the config scripts.) However, the discussion it spawned was informative. The main consideration there was not performance but compatibility with your other libraries. I know on windows, for instance, different run-time libraries are used in each case and all your components must be compiled with the same option. Thankfully, I'm not using windows for this project, so it might not make any difference.<br>
<br>
Eric Preston<br>
<br>
<br>
> Hello,<br>
> I am neither a regular Petsc user nor contributor so preemptive<br>
>apologies if I am completely off the line here.<br>
><br>
>I am not sure if the original poster had hyper-threading in mind when he<br>
>asked about multi-threading, in case that was the idea, I don't think using<br>
>petsc with MKL (HT) is going to give any benefit, I don't think MKL is<br>
>really resource insensitive.<br>
><br>
>Also I wonder what percentage of the code is actually blas/lapack intensive<br>
>to make any significant dent in wall cock?<br>
><br>
>of course +1 to everything else posed above.<br>
><br>
>Cheers,<br>
><br>
>C.S.N<br>
<br>
<br>
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.<br>
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Tue, 15 Mar 2011 22:35:31 +0100<br>
From: Jed Brown <jed@59A2.org><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
To: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
Cc: Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>><br>
Message-ID:<br>
<AANLkTimfHkC8yT9idcTNcv8xOB=<a href="mailto:cM0cLBi8n%2B0vOFAMc@mail.gmail.com">cM0cLBi8n+0vOFAMc@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
On Tue, Mar 15, 2011 at 22:30, Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>>wrote:<br>
<br>
> Regardless of setting the number of threads for MKL or OMP, the MKL<br>
> performance was worse than simply using --download-f-blas-lapack=1.<br>
<br>
<br>
Interesting. Does this statement include using just one thread, perhaps with<br>
a non-threaded MKL? Also, when you used threading, were you putting an MPI<br>
process on every core or were you making sure that you had enough cores for<br>
num_mpi_processes * num_mkl_threads?<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/693c6d6c/attachment-0001.htm" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/693c6d6c/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 15 Mar 2011 17:20:25 -0500<br>
From: Natarajan CS <<a href="mailto:csnataraj@gmail.com">csnataraj@gmail.com</a>><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
To: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
Cc: Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>><br>
Message-ID:<br>
<<a href="mailto:AANLkTinuGyE6Yun%2BKpthjAvoW3VwnEtz%2BkjvD2JGdzS3@mail.gmail.com">AANLkTinuGyE6Yun+KpthjAvoW3VwnEtz+kjvD2JGdzS3@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Thanks Eric and Rob.<br>
<br>
Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread<br>
per core (sequential MKL) is the right thing to do as baseline.<br>
I would think that the performance of #cores = num_mpi_processes *<br>
num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const)<br>
unless some cache effects come into play (Not sure what, I would think the<br>
mkl installation should weed these issues out).<br>
<br>
<br>
P.S :<br>
Out of curiosity have you also tested your app on Nehalem? Any difference<br>
between Nehalem vs Westmere for similar bandwidth?<br>
<br>
On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown <<a href="mailto:jed@59a2.org">jed@59a2.org</a>> wrote:<br>
<br>
> On Tue, Mar 15, 2011 at 22:30, Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>>wrote:<br>
><br>
>> Regardless of setting the number of threads for MKL or OMP, the MKL<br>
>> performance was worse than simply using --download-f-blas-lapack=1.<br>
><br>
><br>
> Interesting. Does this statement include using just one thread, perhaps<br>
> with a non-threaded MKL? Also, when you used threading, were you putting an<br>
> MPI process on every core or were you making sure that you had enough cores<br>
> for num_mpi_processes * num_mkl_threads?<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/96b06f5f/attachment-0001.htm" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/96b06f5f/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Tue, 15 Mar 2011 15:32:44 -0700<br>
From: "Rob Ellis" <Robert.G.Ellis@Shaw.ca><br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
To: "'PETSc users list'" <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
Message-ID: <006301cbe360$e77b63a0$b6722ae0$@G.Ellis@Shaw.ca><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm<br>
currently comparing sequential MKL with --download-f-blas-lapack=1.<br>
<br>
Rob<br>
<br>
<br>
<br>
From: <a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><br>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a>] On Behalf Of Natarajan CS<br>
Sent: Tuesday, March 15, 2011 3:20 PM<br>
To: PETSc users list<br>
Cc: Robert Ellis<br>
Subject: Re: [petsc-users] Building with MKL 10.3<br>
<br>
<br>
<br>
Thanks Eric and Rob.<br>
<br>
Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread<br>
per core (sequential MKL) is the right thing to do as baseline.<br>
I would think that the performance of #cores = num_mpi_processes *<br>
num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const)<br>
unless some cache effects come into play (Not sure what, I would think the<br>
mkl installation should weed these issues out).<br>
<br>
<br>
<br>
P.S :<br>
Out of curiosity have you also tested your app on Nehalem? Any difference<br>
between Nehalem vs Westmere for similar bandwidth?<br>
<br>
On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown <<a href="mailto:jed@59a2.org">jed@59a2.org</a>> wrote:<br>
<br>
On Tue, Mar 15, 2011 at 22:30, Robert Ellis <<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>><br>
wrote:<br>
<br>
Regardless of setting the number of threads for MKL or OMP, the MKL<br>
performance was worse than simply using --download-f-blas-lapack=1.<br>
<br>
<br>
<br>
Interesting. Does this statement include using just one thread, perhaps with<br>
a non-threaded MKL? Also, when you used threading, were you putting an MPI<br>
process on every core or were you making sure that you had enough cores for<br>
num_mpi_processes * num_mkl_threads?<br>
<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/d848575a/attachment-0001.htm" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/d848575a/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
petsc-users mailing list<br>
<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/petsc-users</a><br>
<br>
<br>
End of petsc-users Digest, Vol 27, Issue 33<br>
*******************************************<br>
</blockquote></div><br></div>