<div dir="ltr"><br><br><div class="gmail_quote">On Fri, Mar 6, 2015 at 8:33 PM Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
> On Mar 6, 2015, at 8:22 PM, Dmitry Karpeyev <<a href="mailto:karpeev@mcs.anl.gov" target="_blank">karpeev@mcs.anl.gov</a>> wrote:<br>
><br>
><br>
><br>
> On Fri, Mar 6, 2015 at 6:03 PM Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> > On Mar 6, 2015, at 4:41 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br>
> ><br>
> > Dmitry Karpeyev <<a href="mailto:karpeev@mcs.anl.gov" target="_blank">karpeev@mcs.anl.gov</a>> writes:<br>
> >> This is trickier than it might appear: nonzerostate effectively counts the<br>
> >> global number of nonzeros.<br>
><br>
> No it does not. Note in MatZeroRows_SeqAIJ() when entries are deleted from the matrix we still increase the nonzerostate.<br>
> From MatAssemblyEnd_MPIAIJ():<br>
><br>
> if ((!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) || !((Mat_SeqAIJ*)(aij->A->data))<u></u>->nonew) {<br>
> PetscObjectState state = aij->A->nonzerostate + aij->B->nonzerostate;<br>
<br>
This is ok. So long as the two sub matrix states are always increasing the state of the total matrix will increase.<br>
<br>
> ierr = MPI_Allreduce(&state,&mat-><u></u>nonzerostate,1,MPIU_INT64,MPI_<u></u>SUM,PetscObjectComm((<u></u>PetscObject)mat));CHKERRQ(<u></u>ierr);<br>
> }<br>
><br>
> Barring MatZeroRows MATSEQAIJ matrices aij->A and aij->B will simply increment their nonzerostates on each new nonzero insertion in MatSetVAlues_SeqAIJ(), so the containing MATMPIAIJ M ends up with the total number of nonzeros in its nonzerostate.<br>
><br>
> Now I reset my M. That will blow away aij->A and aij->B.<br>
<br>
They should not be "blown away". They should also be reset and in being reset their nonzerostate will never get smaller.<br></blockquote><div>Okay, this requires implementing MatReset_XXX() that will carefully clean things out for each matrix type, whereas I was using a simple-minded approach of just destroying and recreating the object. Partly because this is more or less the equivalent of the only way to "reset" the matrix now: </div><div>MatSetType(M,othertype);MatSetType(M,oldtype) (*).</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
> I can now insert the same number of nonzeros, but in a different pattern. M will end up with the same nonzerostate as before the reset and confuse the PC, no?<br>
<br>
Why are you reseting the nonzerostate to zero, just don't do that.<br></blockquote><div>I'm not resetting it to anything: nonzerostate will get recomputed from those of A&B using Allreduce. They (A&B), however, will get rebuilt from scratch, if all of M's guts are destroyed and recreated. Even if we insist on writing complicated MatReset_XXX() variants that avoid that, we still need to make sure that (*) above (which will destroy and recreate the inner A&B) doesn't confuse the preconditioner. Or do we simply tell the user not to do that :-)?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
><br>
> This started out as a discussion about MatReset(), but I think this _may_ be a bug we are seeing in one of the elastic contact applications: PCASM tries to rebuild itself with MAT_REUSE_MATRIX when subdomain matrices actually have different numbers of nonzeros. I have to say I haven't ascertained that an inconsistent nonzerostate cases the problem, yet -- reproducible test cases that trigger the problem are still too big to debug.<br>
<br>
It is possible that somewhere the state is not properly handled by being incremented.<br>
<br>
Barry<br>
<br>
><br>
> >> The PC will rebuild if its state is stale, but<br>
> >> it will reuse matrices (e.g., subdomain matrices in PCASM) if nonzerostate<br>
> >> is up to date. This works if the sparsity pattern never drops nonzeros,<br>
> >> but that's no longer true if reset is allowed. I can reset a matrix,<br>
> >> preallocate and assemble it so that the global number of nonzeros will be<br>
> >> the same as before the resetting, but local sparsity patterns will change.<br>
> >> This could happen, for example, when I have moving particles or, less<br>
> >> exotically, when I have elastic contact and nodes move past each other.<br>
> >> That will break PCASM.<br>
><br>
> Just increase the nonzerostate flag by one on a reset (that is there is no reason to ever set it back to zero). Now nonzerostate is monotonically increasing.<br>
><br>
> Barry<br>
><br>
><br>
> Barry<br>
><br>
> ><br>
> > On pretty simple and reliable solution would be to take a cryptographic<br>
> > hash of the row/col arrays. I assume BG is really atrocious at hashing,<br>
> > but is it so bad that this is not viable? (There are several places<br>
> > where we use kinda fragile state counters or trust the user, but hashes<br>
> > would make rebuilding more reliable and transparent.)<br>
><br>
<br>
</blockquote></div></div>