[petsc-users] Varying TAO optimization solve iterations using BLMVM

[Jason Sarich]

Hi Justin,

I don't see anything obviously wrong that would be causing this variation
in iterations due to number of processors. Is it at all feasible to send be
an example code that reproduces the problem (perhaps a smaller version)?
I'm still guessing the problem lies in numerical precision, it would be
nice to find a way to avoid them. I don't think the job scheduling or
compute nodes would affect this.

>  Basically I call TaoSolve at each time level. What I find strange is that
> the number of TAO solve iterations vary at each time level for a given
> number of processors

Just for clarification, do you mean that for a given problem, you run the
same problem several times the only difference being the number of
processors, and that on each time step you get (close to) the same solution
for each run, just with a different number of TAO iterations?

If you run the same problem twice using the same number of processors, is
the output identical? No OpenMP threads or GPU's?


> 2) Sometimes, I get Tao Termination reason of -5, and from what I see from
> the online documentation, it means the number of function evaluations
> exceeds the maximum number of function evaluations. I only get this at
> certain time levels, and it also varies when I change the number of
> processors.


This looks like the same issue, unless the number of iterations is hugely
different (say it converges on some number of processors after 200
iterations, but still hasn't converged after 2000 on a different number).


Jason




On Tue, Jun 23, 2015 at 6:36 PM, Justin Chang <jychang48 at gmail.com> wrote:

>     I was unable to do quad precision or even with 64 bit integers
> because my data files rely on intricate binary files that have been written
> in 32 bit.
>
>  However, I noticed a couple things which are puzzling to me:
>
>  1) I am solving a transient problem using my own backward euler function.
> Basically I call TaoSolve at each time level. What I find strange is that
> the number of TAO solve iterations vary at each time level for a given
> number of processors. The solution is roughly the same when I change the
> number of processors. Any idea why this is happening, or might this have
> more to do with the job scheduling/compute nodes on my HPC machine?
>
>  2) Sometimes, I get Tao Termination reason of -5, and from what I see
> from the online documentation, it means the number of function evaluations
> exceeds the maximum number of function evaluations. I only get this at
> certain time levels, and it also varies when I change the number of
> processors.
>
>  I can understand the number of iterations going down the further in time
> i go (this is due to the nature of my problem), but I am not sure why the
> above two observations are happening. Any thoughts?
>
>  Thanks,
>  Justin
>
> On Fri, Jun 19, 2015 at 11:52 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
>
>> My code sort of requires HDF5 so installing quad precision might be a
>> little difficult. I could try to work around this but that might take some
>> effort.
>>
>>  In the mean time, is there any other potential explanation or
>> alternative to figuring this out?
>>
>>  Thanks,
>> Justin
>>
>>
>> On Thursday, June 18, 2015, Matthew Knepley <knepley at gmail.com> wrote:
>>
>>>  On Thu, Jun 18, 2015 at 1:52 PM, Jason Sarich <jason.sarich at gmail.com>
>>> wrote:
>>>
>>>> BLMVM doesn't use a KSP or preconditioner, it updates using the
>>>> L-BFGS-B formula
>>>>
>>>
>>>  Then this sounds like a bug, unless one of the constants is partition
>>> dependent.
>>>
>>>    Matt
>>>
>>>
>>>>  On Thu, Jun 18, 2015 at 1:45 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>>   On Thu, Jun 18, 2015 at 12:15 PM, Jason Sarich <
>>>>> jason.sarich at gmail.com> wrote:
>>>>>
>>>>>> Hi Justin,
>>>>>>
>>>>>>  I can't tell for sure why this is happening, have you tried using
>>>>>> quad precision to make sure that numerical cutoffs isn't the problem?
>>>>>>
>>>>>>  1 The Hessian being approximate and the resulting implicit
>>>>>> computation is the source of the cutoff, but would not be causing different
>>>>>> convergence rates in infinite precision.
>>>>>>
>>>>>>  2 the local size may affect load balancing but not the resulting
>>>>>> norms/convergence rate.
>>>>>>
>>>>>
>>>>>  This sounds to be like the preconditioner is dependent on the
>>>>> partition. Can you send -tao_view -snes_view
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>>  Jason
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 18, 2015 at 10:44 AM, Justin Chang <jychang48 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>  I solved a transient diffusion across multiple cores using TAO
>>>>>>> BLMVM. When I simulate the same problem but on different numbers of
>>>>>>> processing cores, the number of solve iterations change quite drastically.
>>>>>>> The numerical solution is the same, but these changes are quite vast. I
>>>>>>> attached a PDF showing a comparison between KSP and TAO. KSP remains
>>>>>>> largely invariant with number of processors but TAO (with bounded
>>>>>>> constraints) fluctuates.
>>>>>>>
>>>>>>> My question is, why is this happening? I understand that
>>>>>>> accumulation of numerical round-offs may attribute to this, but the
>>>>>>> differences seem quite vast to me. My initial thought was that
>>>>>>>
>>>>>>>  1) the Hessian is only projected and not explicitly computed,
>>>>>>> which may have something to do with the rate of convergence
>>>>>>>
>>>>>>> 2) local problem size. Certain regions of my domain have different
>>>>>>> number of "violations" which need to be corrected by the bounded
>>>>>>> constraints so the rate of convergence depends on how these regions are
>>>>>>> partitioned?
>>>>>>>
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Justin
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150625/208b0d7a/attachment-0001.html>