On Thu, Apr 22, 2010 at 8:06 AM, Parker, Andrew (UK Filton) <span dir="ltr"><<a href="mailto:Andrew.Parker2@baesystems.com">Andrew.Parker2@baesystems.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word">
<div dir="ltr" align="left"><font face="Arial" color="#0000ff" size="2"><span>Ok,</span></font></div>
<div dir="ltr" align="left"><font face="Arial" color="#0000ff" size="2"><span></span></font> </div>
<div dir="ltr" align="left"><font face="Arial" color="#0000ff" size="2"><span>Cheers guys. I did find that strange, that it was
using AIJ, can I set that in the code rather from an options file, what is the
code equivalent to <font face="Times New Roman" color="#000000" size="3">-mat_type
seqbaij, is it MatSetType or something like that? I'm going to try Jed's
suggestion about error on on first non-preallocated access. Could my error
actually be that the sparsity is correct, but as you note below, the matrix type
allocated with <font face="Arial" size="2">MatSeqBAIJSetPreallocation is of type
AIJ, and in which case, for that matrix it has not be preallocated with the
correct sparsity, or at all, and hence the speed? Or something like
that?? Could the solution possibly just be setting <font face="Times New Roman" size="3">-mat_type seqbaij? In which case I need to
find out how to do that.</font></font></font></span></font></div></div></blockquote><div><br></div><div>1) -mat_type seqbaij is equivalent to MatSetType(A, MATSEQBAIJ), but why hardcode it?</div><div><br></div><div>2) Yes, preallocation only takes place for the given type. You can call more than one</div>
<div> preallocation function.</div><div><br></div><div>3) You have two errors: Wrong matrix type, and then for the second call the matrix type is</div><div> correct but the sparsity is wrong.</div><div><br></div><div>
Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div dir="ltr" align="left"><font face="Arial" color="#0000ff" size="2"><span><font face="Times New Roman" color="#000000" size="3"><font face="Arial" size="2"><font face="Times New Roman" size="3"><span class="Apple-style-span" style="font-family: arial; font-size: small; "> </span></font></font></font></span></font></div>
<div dir="ltr" align="left"><font face="Arial" color="#000000" size="2"><span>Cheers again,</span></font></div>
<div dir="ltr" align="left"><font face="Arial" color="#000000" size="2"><span>Andy</span></font></div><br>
<div lang="en-us" dir="ltr" align="left">
<hr>
<font face="Tahoma" size="2"><b>From:</b> <a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>] <b>On Behalf Of </b>Barry
Smith<br><b>Sent:</b> 22 April 2010 13:59<br><b>To:</b> PETSc users
list<br><b>Subject:</b> Re: [petsc-users] Newbie Question - Really slow -
PetscMemCpy<br></font><br></div>
<div></div><pre style="white-space:normal"> *** WARNING ***<br><br> This message has originated outside your organisation,<br> either from an external partner or the Global Internet. <br> Keep this in mind if you answer this message.<br>
<br></pre><br>
<div>
<div>On Apr 22, 2010, at 7:33 AM, Parker, Andrew (UK Filton) wrote:</div><br>
<blockquote type="cite"><font face="Arial"><font size="2"></font></font>
<div>
<div><span><font face="Arial" size="2">Hi,</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">I'm new to these
postings. On very large cases, regardless of where I stop the debugger
it is always copying data. I've put it into a debugger because on
smaller cases it runs fine, on larger it takes a while (very long time) to get
going. The stack trace always gives something like:</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">memcpy</font></span></div>
<div><span><font face="Arial" size="2">PetscMemCpy</font></span></div>
<div><span><font face="Arial" size="2">MatSetValues_SeqAIJ</font></span></div></div></blockquote>
<div><br></div> It is using SeqAIJ matrices</div>
<div><br>
<blockquote type="cite">
<div>
<div><span><font face="Arial" size="2">MatSetValues</font></span></div>
<div><span><font face="Arial" size="2">my own wrapper to
add values to a location within the matrix.</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">I'm using
Seq_BAIJ. My bets are that I've got the sparsity wrong, or the
preallocation wrong, but I'm not sure why. I know this could be
anything, but has anybody got any thoughts, remember stopping the debugger at
random, regardless of the frequency always gives the
above....</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">I set up the
matrix like this</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">
MatCreate(PETSC_COMM_SELF,&_storage);<br>
MatSetSizes(_storage,PETSC_DECIDE,PETSC_DECIDE,numVars*numLocs,numVars*numLocs);<br>
MatSetFromOptions(_storage);<br>
MatSeqBAIJSetPreallocation(_storage,numVars,PETSC_NULL,sparsityStart);</font></span></div></div></blockquote>
<div><br></div> Unless you use the argument -mat_type seqbaij
this will use SeqAIJ while you preallocate SeqBAIJ.</div>
<div><br>
<blockquote type="cite">
<div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">However, using
this makes zero difference to the speed</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">
MatCreateSeqBAIJ(PETSC_COMM_SELF,<br>
numVars,<br>
numVars*numLocs,<br>
numVars*numLocs,<br>
0, </font></span></div>
<div><span><font face="Arial" size="2"> sparsityStart,</font></span></div>
<div><span><font face="Arial" size="2"> &_storage);</font></span></div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">It is so slow that
it has not even completed one cycle of the
solver....</font></span></div></div></blockquote>
<div><br></div> Your sparseityStart is wrong. Run with -info
and grep the results for malloc and you'll see it is mallocing many times to get
enough space.</div>
<div><br></div>
<div> Barry</div>
<div><br>
<blockquote type="cite">
<div>
<div><span><font face="Arial" size="2"></font></span> </div>
<div><span><font face="Arial" size="2">Cheers
again,<br>Andy</font></span></div><br>********************************************************************<br>This
email and any attachments are confidential to the intended<br>recipient and
may also be privileged. If you are not the intended<br>recipient please delete
it from your system and notify the sender.<br>You should not copy it or use it
for any purpose nor disclose or<br>distribute its contents to any other
person.<br>********************************************************************<br><br></div></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>