<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">We should be able to overlap PetscSFReduce. I will have a look. Thanks.<br clear="all">
<div>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jul 22, 2019 at 5:23 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Junchao,
<div><br>
</div>
<div>I found an issue with PetscSFReduceBegin/End. It seems that we can no longer do (always worked before)</div>
<div><br>
</div>
<div>
<div> ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr);<br>
ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr);</div>
<div> ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr);<br>
ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr);<br>
</div>
<div><br>
</div>
<div>You can reproduce with the currrent master</div>
<div>$ cd $PETSC_DIR/src/ksp/ksp/examples/tutorials</div>
<div>$ make ex71</div>
<div>$ mpiexec -n 4 ./ex71 -pde_type Elasticity -cells 7,9 -dim 2 -pc_bddc_levels 1 -pc_bddc_coarsening_ratio 2 -ksp_error_if_not_converged -pc_bddc_monolithic -pc_bddc_use_faces -pc_bddc_coarse_pc_bddc_corner_selection -pc_bddc_coarse_l1_pc_bddc_corner_selection
-mat_partitioning_type average -pc_bddc_coarse_pc_bddc_use_deluxe_scaling -pc_bddc_coarse_sub_schurs_mat_solver_type petsc<br>
</div>
<div><br>
</div>
<div>The attached patch shows that if we instead do</div>
<div><br>
</div>
<div> ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr);</div>
<div> ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf,root,MPI_MAX);CHKERRQ(ierr);<br>
</div>
<div> ierr = PetscSFReduceBegin(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr);</div>
<div> ierr = PetscSFReduceEnd(sf,MPIU_INT,leaf2,root2,MPI_MAX);CHKERRQ(ierr);<br>
</div>
<div><br>
</div>
<div>everything works properly. </div>
I think the issue is with the SF GetPack mechanism, but I couldn't figure out where.</div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr" class="gmail-m_917648182967337374gmail_signature">Stefano</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>