<div dir="ltr"><div dir="ltr"><div dir="ltr">Based on Barry's encouragement I spent the extra time to get back to v3.10.2 and this code seems to work:<div><br></div><div><div>int mpierr;</div><div>MPI_Comm mpiCommRaw;</div><div>int tag = 0;</div><div>mpierr = MPI_Comm_dup(MPI_COMM_SELF, &mpiCommRaw); if(mpierr != MPI_SUCCESS) throw PSI::Exception("MPI ERROR: MPI_Comm_dup");</div><div><br></div><div>MPI_Comm mpiComm;</div><div>ierr = PetscCommDuplicate(mpiCommRaw, &mpiComm, &tag); CHKERRABORT(PETSC_COMM_SELF, ierr);</div><div><br></div><div>Tao tao;</div><div>ierr = TaoCreate(mpiComm, &tao); CHKERRABORT(mpiComm, ierr);</div><div><br></div><div>...</div><div>ierr = TaoSolve(tao); CHKERRABORT(mpiComm, ierr);</div><div>...</div><div><br></div><div>ierr = TaoDestroy(&tao); CHKERRABORT(mpiComm, ierr);</div><div><br></div><div>ierr = PetscCommDestroy(&mpiComm); CHKERRABORT(PETSC_COMM_SELF, ierr);</div><div><br></div><div>mpierr = MPI_Comm_free(&mpiCommRaw); if(mpierr != MPI_SUCCESS) throw PSI::Exception("MPI ERROR: MPI_Comm_free");</div></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Dec 20, 2018 at 4:06 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> On Dec 20, 2018, at 2:24 PM, Krzysztof Kamieniecki via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> Hi Alp,<br>
> <br>
> Thanks! This worked, I reverted back to v3.9.4 and after removing the monitors (which caused an error in PetscViewerASCIIPopTab) it seems to be passing tests for now.<br>
> <br>
> (For the future peanut gallery) I misread what PetscCommDuplicate does, it does not duplicate Petsc communicators that already "wrap" MPI communicators, so I may look into MPI and creating a completely independent MPI_Comm for each thread.<br>
<br>
Yes, this should work.<br>
> <br>
> Best Regards,<br>
> Krys<br>
> <br>
> On Thu, Dec 20, 2018 at 12:16 PM Dener, Alp <<a href="mailto:adener@anl.gov" target="_blank">adener@anl.gov</a>> wrote:<br>
> Hi Krys,<br>
> <br>
>> On Dec 20, 2018, at 10:59 AM, Krzysztof Kamieniecki via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
>> <br>
>> That example seems to have critical sections around certain Vec calls, and it looks like my problem occurs in VecDotBegin/VecDotEnd which is called by TAO/BLMVM.<br>
> <br>
> The quasi-Newton matrix objects in BLMVM have asynchronous dot products in the matrix-free forward and inverse product formulations. This is a relatively recent performance optimization. If avoiding this split phase communication would solve the problem, and you don’t need other recent PETSc features, you could revert to 3.9 and use the old version of BLMVM that will use straight VecDot operations instead.<br>
> <br>
> Unfortunately I don’t know enough about multithreading to definitively say whether that will actually solve the problem or not. Other members of the community can probably provide a more complete answer on that.<br>
> <br>
>> <br>
>> I assume PetscSplitReductionGet is pulling the PetscSplitReduction for PETSC_COMM_SELF which is shared across the whole process?<br>
>> <br>
>> I tried PetscCommDuplicate/PetscCommDestroy but that does not seem to help.<br>
>> <br>
>> PetscErrorCode VecDotBegin(Vec x,Vec y,PetscScalar *result)<br>
>> {<br>
>> PetscErrorCode ierr;<br>
>> PetscSplitReduction *sr;<br>
>> MPI_Comm comm;<br>
>> <br>
>> PetscFunctionBegin;<br>
>> PetscValidHeaderSpecific(x,VEC_CLASSID,1);<br>
>> PetscValidHeaderSpecific(y,VEC_CLASSID,1);<br>
>> ierr = PetscObjectGetComm((PetscObject)x,&comm);CHKERRQ(ierr);<br>
>> ierr = PetscSplitReductionGet(comm,&sr);CHKERRQ(ierr);<br>
>> if (sr->state != STATE_BEGIN) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ORDER,"Called before all VecxxxEnd() called");<br>
>> if (sr->numopsbegin >= sr->maxops) {<br>
>> ierr = PetscSplitReductionExtend(sr);CHKERRQ(ierr);<br>
>> }<br>
>> sr->reducetype[sr->numopsbegin] = PETSC_SR_REDUCE_SUM;<br>
>> sr->invecs[sr->numopsbegin] = (void*)x;<br>
>> if (!x->ops->dot_local) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Vector does not suppport local dots");<br>
>> ierr = PetscLogEventBegin(VEC_ReduceArithmetic,0,0,0,0);CHKERRQ(ierr);<br>
>> ierr = (*x->ops->dot_local)(x,y,sr->lvalues+sr->numopsbegin++);CHKERRQ(ierr);<br>
>> ierr = PetscLogEventEnd(VEC_ReduceArithmetic,0,0,0,0);CHKERRQ(ierr);<br>
>> PetscFunctionReturn(0);<br>
>> }<br>
>> <br>
>> <br>
>> <br>
>> <br>
>> On Thu, Dec 20, 2018 at 11:26 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
>> <br>
>> The code src/ksp/ksp/examples/tutorials/ex61f.F90 demonstrates working with multiple threads each managing their own collection of PETSc objects. Hope this helps.<br>
>> <br>
>> Barry<br>
>> <br>
>> <br>
>> > On Dec 20, 2018, at 9:28 AM, Krzysztof Kamieniecki via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
>> > <br>
>> > Hello All,<br>
>> > <br>
>> > I have an embarrassingly parallel problem that I would like to use TAO on, is there some way to do this with threads as opposed to multiple processes?<br>
>> > <br>
>> > I compiled PETSc with the following flags<br>
>> > ./configure \<br>
>> > --prefix=${DEP_INSTALL_DIR} \<br>
>> > --with-threadsafety --with-log=0 --download-concurrencykit \<br>
>> > --with-openblas=1 \<br>
>> > --with-openblas-dir=${DEP_INSTALL_DIR} \<br>
>> > --with-mpi=0 \<br>
>> > --with-shared=0 \<br>
>> > --with-debugging=0 COPTFLAGS='-O3' CXXOPTFLAGS='-O3' FOPTFLAGS='-O3' <br>
>> > <br>
>> > When I run TAO in multiple threads I get the error "Called VecxxxEnd() in a different order or with a different vector than VecxxxBegin()"<br>
>> > <br>
>> > Thanks,<br>
>> > Krys<br>
>> > <br>
>> <br>
> <br>
> —<br>
> Alp<br>
> <br>
<br>
</blockquote></div>