<div dir="ltr"><div>Satish,<br></div>I have a MR !2688 to fix it. How to revert your revert to get the MR actually tested?<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 3, 2020 at 9:11 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><a href="https://gitlab.com/petsc/petsc/pipelines/132414153/builds" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/pipelines/132414153/builds</a><br>
<br>
this pipeline had 1 rerun for linux-cuda-double and 5 reruns for linux-c-exodus-dbg<br>
<br>
I've reverted 2655 for now. We can revert this once the issue is resolved.<br>
<br>
Satish<br>
<br>
On Thu, 2 Apr 2020, Satish Balay via petsc-dev wrote:<br>
<br>
> Perhaps we should revert this merge - so that the CI is stabilized for now. <br>
> <br>
> Satish<br>
> <br>
> On Thu, 2 Apr 2020, Junchao Zhang wrote:<br>
> <br>
> > Seems caused by MR 2655<br>
> > <<a href="https://gitlab.com/petsc/petsc/-/merge_requests/2655" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/2655</a>>. I reverted it and<br>
> > tested in CI several times and the error did not appear. Let's assume the<br>
> > MR has a bug. I am looking into it.<br>
> > <br>
> > --Junchao Zhang<br>
> > <br>
> > <br>
> > On Thu, Apr 2, 2020 at 10:58 AM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> > <br>
> > > That was a different error. This does keep coming up occasionally.<br>
> > > <a href="https://gitlab.com/petsc/petsc/-/issues/360#note_250063306" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/issues/360#note_250063306</a><br>
> > ><br>
> > > The current issue is:<br>
> > > <a href="https://gitlab.com/petsc/petsc/-/issues/360#note_314185490" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/issues/360#note_314185490</a><br>
> > ><br>
> > > Satish<br>
> > ><br>
> > ><br>
> > > On Thu, 2 Apr 2020, Karl Rupp wrote:<br>
> > ><br>
> > > > The fluctuations in this example have been fixed a few months ago; the<br>
> > > issue<br>
> > > > was the use of multiple streams instead of a single one. Maybe<br>
> > > additional CUDA<br>
> > > > streams have been reintroduced recently?<br>
> > > ><br>
> > > > Best regards,<br>
> > > > Karli<br>
> > > ><br>
> > > ><br>
> > > > On 4/2/20 5:02 AM, Junchao Zhang wrote:<br>
> > > > > I could not reproduce it locally. Even in the CI, it is random.<br>
> > > > ><br>
> > > > > --Junchao Zhang<br>
> > > > ><br>
> > > > ><br>
> > > > > On Wed, Apr 1, 2020 at 7:47 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a><br>
> > > > > <mailto:<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>> wrote:<br>
> > > > ><br>
> > > > > I saw Satish talking about this on the CI Tracker MR.<br>
> > > > ><br>
> > > > > Matt<br>
> > > > ><br>
> > > > > On Wed, Apr 1, 2020 at 8:36 PM Lisandro Dalcin <<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a><br>
> > > > > <mailto:<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>>> wrote:<br>
> > > > ><br>
> > > > > Well, my request will not fix the problem:<br>
> > > > > <a href="https://gitlab.com/petsc/petsc/-/jobs/495147366#L5231" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/jobs/495147366#L5231</a><br>
> > > > ><br>
> > > > > On Thu, 2 Apr 2020 at 03:26, Lisandro Dalcin <<br>
> > > <a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a><br>
> > > > > <mailto:<a href="mailto:dalcinl@gmail.com" target="_blank">dalcinl@gmail.com</a>>> wrote:<br>
> > > > ><br>
> > > > > Can anyone messing with CPUs please update test<br>
> > > > > snes_tutorials-ex19_cuda_1 to use -ksp_monitor_short and<br>
> > > > > update its output with REPLACE=1 ?<br>
> > > > ><br>
> > > > > Please do it in maint, or cherry-pick if already fixed in<br>
> > > > > master.<br>
> > > > ><br>
> > > > > Regards,<br>
> > > > ><br>
> > > > > --<br>
> > > > > Lisandro Dalcin<br>
> > > > > ============<br>
> > > > > Research Scientist<br>
> > > > > Extreme Computing Research Center (ECRC)<br>
> > > > > King Abdullah University of Science and Technology (KAUST)<br>
> > > > > <a href="http://ecrc.kaust.edu.sa/" rel="noreferrer" target="_blank">http://ecrc.kaust.edu.sa/</a><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > --<br>
> > > > > Lisandro Dalcin<br>
> > > > > ============<br>
> > > > > Research Scientist<br>
> > > > > Extreme Computing Research Center (ECRC)<br>
> > > > > King Abdullah University of Science and Technology (KAUST)<br>
> > > > > <a href="http://ecrc.kaust.edu.sa/" rel="noreferrer" target="_blank">http://ecrc.kaust.edu.sa/</a><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > --<br>
> > > > > What most experimenters take for granted before they begin their<br>
> > > > > experiments is infinitely more interesting than any results to<br>
> > > which<br>
> > > > > their experiments lead.<br>
> > > > > -- Norbert Wiener<br>
> > > > ><br>
> > > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> > > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> > <br>
> <br>
<br>
</blockquote></div>