<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Aug 1, 2016 at 9:36 AM, Klaij, Christiaan <span dir="ltr"><<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p></p>
<p>Matt,</p>
<p><br>
</p>
<p>1) great!<br>
</p>
<p><br>
</p>
<p>2) ??? that's precisely why I paste the output of "cat mattry.F90" in the emails, so you have a small example that produces the errors I mention. Now I'm also attaching it to this email.</p></div></blockquote><div>Okay, I have gone through it. You are correct that it is completely broken.</div><div><br></div><div>The way that MatNest currently works is that it trys to use L2G mappings from individual blocks</div><div>and then builds a composite L2G map for the whole matrix. This is obviously incompatible with</div><div>the primary use case, and should be changed to break up the full L2G into one for each block.<br></div><div><br></div><div>Jed, can you fix this? I am not sure I know enough about how Nest works.</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><p><span style="font-size:12pt">Thanks,</span></p>
<p>Chris<br>
</p>
<div style="color:rgb(33,33,33)"></div>
<br>
<font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">dr. ir.</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Christiaan</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Klaij</font><font size="3" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">CFD Researcher</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Research & Development</font><br>
<font style="font-family:Verdana;font-size:9pt;color:#7d7d7d;font-weight:bold">MARIN</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d"> | T
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d"><a href="tel:%2B31%20317%2049%2033%2044" value="+31317493344" target="_blank">+31 317 49 33 44</a></font><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="mailto:C.Klaij@marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">C.Klaij@marin.nl</span></a></span><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="http://www.marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">www.marin.nl</span></a></span><br>
<br>
<a href="https://www.linkedin.com/company/marin" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:imagedaaefa.PNG@9f2a039a.49a8f7e1" alt="LinkedIn"></a> <a href="http://www.youtube.com/marinmultimedia" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:imagedf4413.PNG@982909ee.48b44282" alt="YouTube"></a> <a href="https://twitter.com/MARIN_nieuws" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:imagee68916.PNG@53464e85.4c903b5e" alt="Twitter"></a> <a href="https://www.facebook.com/marin.wageningen" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:image049e4b.PNG@620146c4.43893aa7" alt="Facebook"></a><br>
<font style="FONT-FAMILY:Verdana;FONT-SIZE:8pt" color="#7d7d7d" size="2" face="Verdana">MARIN news:
<font style="font-family:Verdana;font-size:8pt"><span style="font-family:Verdana;font-size:8pt"><a href="http://www.marin.nl/web/News/News-items/Vice-Admiraal-De-Waard-maakt-virtuele-proefvaart-op-MARINs-FSSS-Dutch-only.htm" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">Vice
Admiraal De Waard maakt virtuele proefvaart op MARIN’s FSSS (Dutch only)</span></a></span></font></font>
<p></p>
<p></p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Monday, August 01, 2016 4:15 PM<br>
<b>To:</b> Klaij, Christiaan<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] block matrix without MatCreateNest</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Mon, Aug 1, 2016 at 8:59 AM, Klaij, Christiaan <span dir="ltr">
<<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Matt,<br>
<br>
Thanks for your suggestions. Here's the outcome:<br>
<br>
1) without the "if type=nest" protection, I get a "Cannot<br>
locate function MatNestSetSubMats_C" error when using<br>
type mpiaij, see below.<br>
</blockquote>
<div><br>
</div>
<div>That is a bug. It should be using PetscTryMethod() there, not PetscUseMethod(). We will fix it. That way</div>
<div>it can be called for any matrix type, which is the intention.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) with the isg in a proper array, I get the same "Invalid<br>
Pointer to Object" error, see below.<br>
</blockquote>
<div><br>
</div>
<div>Can you send a small example that we can run? It obviously should work this way.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Chris<br>
<br>
<br>
$ cat mattry.F90<br>
program mattry<br>
<br>
use petscksp<br>
implicit none<br>
#include <petsc/finclude/petsckspdef.h><br>
<br>
PetscInt :: n=4 ! setting 4 cells per process<br>
<br>
PetscErrorCode :: ierr<br>
PetscInt :: size,rank,i<br>
Mat :: A,A02<br>
MatType :: type<br>
IS :: isg(3)<br>
IS :: isl(3)<br>
ISLocalToGlobalMapping :: map<br>
<br>
integer, allocatable, dimension(:) :: idx<br>
<br>
call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr)<br>
call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr)<br>
call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr);CHKERRQ(ierr)<br>
<br>
! local index sets for 3 fields<br>
allocate(idx(n))<br>
idx=(/ (i-1, i=1,n) /)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx,PETSC_COPY_VALUES,isl(1),ierr);CHKERRQ(ierr)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx+n,PETSC_COPY_VALUES,isl(2),ierr);CHKERRQ(ierr)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx+2*n,PETSC_COPY_VALUES,isl(3),ierr);CHKERRQ(ierr)<br>
! call ISView(isl3,PETSC_VIEWER_STDOUT_WORLD,ierr); CHKERRQ(ierr)<br>
deallocate(idx)<br>
<br>
! global index sets for 3 fields<br>
allocate(idx(n))<br>
idx=(/ (i-1+rank*3*n, i=1,n) /)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx,PETSC_COPY_VALUES,isg(1),ierr);CHKERRQ(ierr)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx+n,PETSC_COPY_VALUES,isg(2),ierr); CHKERRQ(ierr)<br>
call ISCreateGeneral(PETSC_COMM_WORLD,n,idx+2*n,PETSC_COPY_VALUES,isg(3),ierr); CHKERRQ(ierr)<br>
! call ISView(isg3,PETSC_VIEWER_STDOUT_WORLD,ierr); CHKERRQ(ierr)<br>
deallocate(idx)<br>
<br>
! local-to-global mapping<br>
allocate(idx(3*n))<br>
idx=(/ (i-1+rank*3*n, i=1,3*n) /)<br>
call ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD,1,3*n,idx,PETSC_COPY_VALUES,map,ierr); CHKERRQ(ierr)<br>
! call ISLocalToGlobalMappingView(map,PETSC_VIEWER_STDOUT_WORLD,ierr); CHKERRQ(ierr)<br>
deallocate(idx)<br>
<br>
! create the 3-by-3 block matrix<br>
call MatCreate(PETSC_COMM_WORLD,A,ierr); CHKERRQ(ierr)<br>
call MatSetSizes(A,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,ierr); CHKERRQ(ierr)<br>
call MatSetUp(A,ierr); CHKERRQ(ierr)<br>
call MatSetOptionsPrefix(A,"A_",ierr); CHKERRQ(ierr)<br>
call MatSetLocalToGlobalMapping(A,map,map,ierr); CHKERRQ(ierr)<br>
call MatSetFromOptions(A,ierr); CHKERRQ(ierr)<br>
<br>
! setup nest<br>
call MatNestSetSubMats(A,3,isg,3,isg,PETSC_NULL_OBJECT,ierr); CHKERRQ(ierr)<br>
<br>
! set diagonal of block A02 to 0.65<br>
call MatGetLocalSubmatrix(A,isl(1),isl(3),A02,ierr); CHKERRQ(ierr)<br>
do i=1,n<br>
call MatSetValuesLocal(A02,1,i-1,1,i-1,0.65d0,INSERT_VALUES,ierr); CHKERRQ(ierr)<br>
end do<br>
call MatRestoreLocalSubMatrix(A,isl(1),isl(3),A02,ierr); CHKERRQ(ierr)<br>
call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr); CHKERRQ(ierr)<br>
call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr); CHKERRQ(ierr)<br>
<br>
! verify<br>
call MatGetSubmatrix(A,isg(1),isg(3),MAT_INITIAL_MATRIX,A02,ierr); CHKERRQ(ierr)<br>
call MatView(A02,PETSC_VIEWER_STDOUT_WORLD,ierr);CHKERRQ(ierr)<br>
<br>
call PetscFinalize(ierr)<br>
<br>
end program mattry<br>
<br>
<br>
$ mpiexec -n 2 ./mattry -A_mat_type mpiaij<br>
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[0]PETSC ERROR: No support for this operation for this object type<br>
[0]PETSC ERROR: Cannot locate function MatNestSetSubMats_C in object<br>
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.7.3, Jul, 24, 2016<br>
[0]PETSC ERROR: ./mattry
on a linux_64bit_debug named lin0322.marin.local by cklaij Mon Aug 1 15:42:10 2016<br>
[0]PETSC ERROR: Configure options --with-mpi-dir=/home/cklaij/ReFRESCO/Dev/trunk/Libs/install/openmpi/1.8.7 --with-clanguage=c++ --with-x=1 --with-debugging=1 --with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl --with-shared-libraries=0<br>
[0]PETSC ERROR: #1 MatNestSetSubMats() line 1105 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
--------------------------------------------------------------------------<br>
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>
with errorcode 56.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
--------------------------------------------------------------------------<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
[1]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">
http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[1]PETSC ERROR: likely location of problem given in stack below<br>
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[1]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[1]PETSC ERROR: is given.<br>
[1]PETSC ERROR: [1] PetscSleep line 35 /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/sys/utils/psleep.c<br>
[1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 193 /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/sys/error/errtrace.c<br>
[1]PETSC ERROR: [1] PetscError line 364 /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/sys/error/err.c<br>
[1]PETSC ERROR: [1] MatNestSetSubMats line 1092 /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[1]PETSC ERROR: Signal received<br>
[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.7.3, Jul, 24, 2016<br>
[1]PETSC ERROR: ./mattry
on a linux_64bit_debug named lin0322.marin.local by cklaij Mon Aug 1 15:42:10 2016<br>
[1]PETSC ERROR: Configure options --with-mpi-dir=/home/cklaij/ReFRESCO/Dev/trunk/Libs/install/openmpi/1.8.7 --with-clanguage=c++ --with-x=1 --with-debugging=1 --with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl --with-shared-libraries=0<br>
[1]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>
[lin0322.marin.local:19455] 1 more process has sent help message help-mpi-api.txt / mpi-abort<br>
[lin0322.marin.local:19455] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages<br>
<br>
<br>
$ mpiexec -n 2 ./mattry -A_mat_type nest<br>
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[0]PETSC ERROR: Corrupt argument: <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1<br>
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[1]PETSC ERROR: Corrupt argument: <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
[1]PETSC ERROR: Invalid Pointer to Object: Parameter # 1<br>
[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.7.3, Jul, 24, 2016<br>
[1]PETSC ERROR: ./mattry
on a linux_64bit_debug named lin0322.marin.local by cklaij Mon Aug 1 15:42:25 2016<br>
[1]PETSC ERROR: Configure options --with-mpi-dir=/home/cklaij/ReFRESCO/Dev/trunk/Libs/install/openmpi/1.8.7 --with-clanguage=c++ --with-x=1 --with-debugging=1 --with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl --with-shared-libraries=0<br>
[1]PETSC ERROR: #1 PetscObjectReference() line 534 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/sys/objects/inherit.c<br>
[1]PETSC ERROR: #2 MatNestSetSubMats_Nest() line 1042 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
[1]PETSC ERROR: #3 MatNestSetSubMats() line 1105 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: Petsc Release Version 3.7.3, Jul, 24, 2016<br>
[0]PETSC ERROR: ./mattry
on a linux_64bit_debug named lin0322.marin.local by cklaij Mon Aug 1 15:42:25 2016<br>
[0]PETSC ERROR: Configure options --with-mpi-dir=/home/cklaij/ReFRESCO/Dev/trunk/Libs/install/openmpi/1.8.7 --with-clanguage=c++ --with-x=1 --with-debugging=1 --with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl --with-shared-libraries=0<br>
[0]PETSC ERROR: #1 PetscObjectReference() line 534 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/sys/objects/inherit.c<br>
[0]PETSC ERROR: #2 MatNestSetSubMats_Nest() line 1042 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
[0]PETSC ERROR: #3 MatNestSetSubMats() line 1105 in /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc/3.7.3-dbg/src/mat/impls/nest/matnest.c<br>
--------------------------------------------------------------------------<br>
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD<br>
with errorcode 64.<br>
<br>
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
You may or may not see output from other processes, depending on<br>
exactly when Open MPI kills them.<br>
--------------------------------------------------------------------------<br>
[lin0322.marin.local:19465] 1 more process has sent help message help-mpi-api.txt / mpi-abort<br>
[lin0322.marin.local:19465] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages<br>
$<br>
<br>
<br>
dr. ir. Christiaan Klaij | CFD Researcher | Research & Development<br>
MARIN | T <a href="tel:%2B31%20317%2049%2033%2044" value="+31317493344" target="_blank">+31 317 49 33 44</a> | mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a> |
<a href="http://www.marin.nl" rel="noreferrer" target="_blank">http://www.marin.nl</a><br>
<br>
MARIN news: <a href="http://www.marin.nl/web/News/News-items/Ship-design-in-EU-project-Holiship.htm" rel="noreferrer" target="_blank">
http://www.marin.nl/web/News/News-items/Ship-design-in-EU-project-Holiship.htm</a><br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</font></span></div>
</div>
</div>
</div>
<u></u>
<p></p>
<u></u><u></u><u></u>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>