<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Mar 23, 2016 at 7:11 PM, Steena Monteiro <span dir="ltr"><<a href="mailto:steena.hpc@gmail.com" target="_blank">steena.hpc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks, Barry. The block size compatibility was a probe to investigate an error arising when trying to assign unequal numbers of rows across two MPI ranks.<div><br><div>To re-visit the initial question, what is going wrong when I try to divide rows unequally across MPI ranks using MatSetSize?</div></div></div></blockquote><div><br></div><div>We definitely require that the blocksize divide the local size. This looks like a problem with our checks.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>For matrix A with rows=cols=1,139,905 and block size = 2, <br></div><div><p>rank 0 gets 400000 rows and rank 1, 739905 rows</p>
<p>if (!rank) {<br></p>
<p> ierr = MatSetSizes(A, 400000, PETSC_DECIDE, 1139905,1139905);CHKERRQ(ierr);</p>
<p> } </p>
<p>else {</p>
<p> ierr = MatSetSizes(A, 739905, PETSC_DECIDE, 1139905,1139905);CHKERRQ(ierr);</p>
<p>}</p><p>MatMult (A,x,y);</p>
<p><br></p>
<p>/************************************/</p>
<p>Error message:<br></p>
<p>[1]PETSC ERROR: [0]PETSC ERROR: No support for this operation for this object type<br></p>
<p>Cannot change/reset row sizes to 400000 local 1139906 global after previously setting them to 400000 local 1139905 global</p>
<p>[1]PETSC ERROR: [0]PETSC ERROR: Cannot change/reset row sizes to 739905 local 1139906 global after previously setting them to 739905 local 1139905 global<br></p></div><div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 23 March 2016 at 15:46, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
The baij and sbaij MatLoad() do automatically pad the matrix with rows/columns of the identity to make it divisible by the block size. This is why you are seeing what you are seeing.<br>
<br>
Barry<br>
<br>
Is this a good idea? Maybe not<br>
<div><div><br>
<br>
> On Mar 22, 2016, at 8:59 PM, Steena Monteiro <<a href="mailto:steena.hpc@gmail.com" target="_blank">steena.hpc@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
><br>
> So, are you saying that<br>
><br>
> 1) You have a matrix with odd total dimension<br>
><br>
> 2) You set the block size of the initial matrix to 2<br>
><br>
> 3) You load the matrix<br>
><br>
> and there is no error? Can you make a simple example with a matrix of size 5?<br>
> I can put in the relevant error checking.<br>
><br>
><br>
><br>
> Hi Matt,<br>
><br>
> Thank you for the suggestion.<br>
><br>
> I used cage3, a 5x5 matrix from the UFL collection, converted it to binary, and tested the code for block sizes 1 through 7. I added printfs inside all the MatMult_SeqBAIJ_*s in baij2.c and also logged some counts (blocked rows and blocks). The counts make sense if the matrix is being padded somewhere to accommodate for block sizes that are not completely divisible by matrix dimensions.<br>
><br>
> surface86@monteiro:./rvector-petsctrain-seqbaij -fin cage3.dat -matload_block_size 2<br>
> Inside SeqBAIJ_2<br>
><br>
> surface86@monteiro:./rvector-petsctrain-seqbaij -fin cage3.dat -matload_block_size 3<br>
> Inside MatMult_SeqBAIJ_3<br>
> ...<br>
> ...<br>
><br>
> surface86@monteiro:./rvector-petsctrain-seqbaij -fin cage3.dat -matload_block_size 7<br>
> Inside MatMult_SeqBAIJ_7<br>
><br>
> Table for different block sizes listing corresponding number of blocked rows and number of blocks inside the rows for cage3.<br>
><br>
><br>
> Block size No. of blocked rows No. of nnz blocks in each blocked row.<br>
> 1 5 5,3,3,4,4<br>
> 2 3 3,3,3<br>
> 3 2 2,2<br>
> 4 2 2,2<br>
> 5 1 1<br>
> 6 1 1<br>
> 7 1 1<br>
><br>
> I am attaching cage3.dat and cage3.mtx.<br>
><br>
> Thanks,<br>
> Steena<br>
><br>
><br>
><br>
><br>
><br>
><br>
> ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr);<br>
><br>
><br>
> ierr = VecSetRandom(x,NULL); CHKERRQ(ierr);<br>
> ierr = VecSet(y,zero); CHKERRQ(ierr);<br>
> ierr = MatMult(A,x,y); CHKERRQ(ierr);<br>
><br>
><br>
> ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr);<br>
> ierr = MatDestroy(&A);CHKERRQ(ierr);<br>
> ierr = VecDestroy(&x);CHKERRQ(ierr);<br>
> ierr = VecDestroy(&y);CHKERRQ(ierr);<br>
><br>
> Thanks,<br>
> Steena<br>
><br>
><br>
> On 15 March 2016 at 09:15, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Tue, Mar 15, 2016 at 11:04 AM, Steena Monteiro <<a href="mailto:steena.hpc@gmail.com" target="_blank">steena.hpc@gmail.com</a>> wrote:<br>
> I pass a binary, matrix data file at the command line and load it into the matrix:<br>
><br>
> PetscInitialize(&argc,&args,(char*)0,help);<br>
> ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);<br>
><br>
> /* converted mtx to dat file*/<br>
> ierr = PetscOptionsGetString(NULL,"-f",file,PETSC_MAX_PATH_LEN,&flg);<br>
> CHKERRQ(ierr);<br>
><br>
> if (!flg) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_USER,"specify matrix dat file with -f");<br>
><br>
> /* Load matrices */<br>
> ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,file,FILE_MODE_READ,&fd);CHKERRQ(ierr);<br>
> ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,file,FILE_MODE_READ,&fd);CHKERRQ(ierr);<br>
> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);<br>
> ierr = MatSetFromOptions(A);CHKERRQ(ierr);<br>
><br>
> Nothing above loads a matrix. Do you also call MatLoad()?<br>
><br>
> Matt<br>
><br>
> Thanks,<br>
> Steena<br>
><br>
> On 15 March 2016 at 08:58, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Tue, Mar 15, 2016 at 10:54 AM, Steena Monteiro <<a href="mailto:steena.hpc@gmail.com" target="_blank">steena.hpc@gmail.com</a>> wrote:<br>
> Thank you, Dave.<br>
><br>
> Matt: I understand the inconsistency but MatMult with non divisible block sizes (here, 2) does not throw any errors and fail, when MatSetSize is commented out. Implying that 1139905 global size does work with block size 2.<br>
><br>
> If you comment out MatSetSize(), how does it know what size the Mat is?<br>
><br>
> Matt<br>
><br>
> On 15 March 2016 at 00:12, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
><br>
> On 15 March 2016 at 04:46, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Mon, Mar 14, 2016 at 10:05 PM, Steena Monteiro <<a href="mailto:steena.hpc@gmail.com" target="_blank">steena.hpc@gmail.com</a>> wrote:<br>
> Hello,<br>
><br>
> I am having difficulty getting MatSetSize to work prior to using MatMult.<br>
><br>
> For matrix A with rows=cols=1,139,905 and block size = 2,<br>
><br>
> It is inconsistent to have a row/col size that is not divisible by the block size.<br>
><br>
><br>
> To be honest, I don't think the error message being thrown clearly indicates what the actual problem is (hence the email from Steena). What about<br>
><br>
> "Cannot change/reset row sizes to 400000 local 1139906 global after previously setting them to 400000 local 1139905 global. Local and global sizes must be divisible by the block size"<br>
><br>
><br>
> Matt<br>
><br>
> rank 0 gets 400000 rows and rank 1 739905 rows, like so:<br>
><br>
> /*Matrix setup*/<br>
><br>
> ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,file,FILE_MODE_READ,&fd);<br>
> ierr = MatCreate(PETSC_COMM_WORLD,&A);<br>
> ierr = MatSetFromOptions(A);<br>
> ierr = MatSetType(A,MATBAIJ);<br>
> ierr = MatSetBlockSize(A,2);<br>
><br>
> /*Unequal row assignment*/<br>
><br>
> if (!rank) {<br>
> ierr = MatSetSizes(A, 400000, PETSC_DECIDE, 1139905,1139905);CHKERRQ(ierr);<br>
> }<br>
> else {<br>
> ierr = MatSetSizes(A, 739905, PETSC_DECIDE, 1139905,1139905);CHKERRQ(ierr);<br>
> }<br>
><br>
> MatMult (A,x,y);<br>
><br>
> /************************************/<br>
><br>
> Error message:<br>
><br>
> 1]PETSC ERROR: [0]PETSC ERROR: No support for this operation for this object type<br>
> Cannot change/reset row sizes to 400000 local 1139906 global after previously setting them to 400000 local 1139905 global<br>
><br>
> [1]PETSC ERROR: [0]PETSC ERROR: Cannot change/reset row sizes to 739905 local 1139906 global after previously setting them to 739905 local 1139905 global<br>
><br>
> -Without messing with row assignment, MatMult works fine on this matrix for block size = 2, presumably because an extra padded row is automatically added to facilitate blocking.<br>
><br>
> -The above code snippet works well for block size = 1.<br>
><br>
> Is it possible to do unequal row distribution while using blocking?<br>
><br>
> Thank you for any advice.<br>
><br>
> -Steena<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
</div></div>> <cage3.dat><cage3.mtx><br>
<br>
</blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>