Thanks. Following this conversation closely, i have already built petsc with --with-debugging=0 and found that then thing work better there. Until there is a ultimate fix I'm perfectly happy with that. <br><br>Even on my cluster, I can now run the program for 2,4,8 cores. But there is always a catch... with 10 cores, GMRES breaks down! (see below) Sorry to bother you with this, it seems unrelated at first sight. Should I issue a new thread for this?<br>
<br>Regards, Leo<br><br>[5]PETSC ERROR: --------------------- Error Message ------------------------------------<br>[5]PETSC ERROR: Nonconforming object sizes!<br>[5]PETSC ERROR: Mat mat,Vec y: local dim 210161 213282!<br>
[5]PETSC ERROR: ------------------------------------------------------------------------<br>[5]PETSC ERROR: Petsc Development HG revision: 6da7d8c5aaa81099861622f4e6bf1087652253a7 HG Date: Fri Sep 03 15:06:40 2010 +0200<br>
[5]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[5]PETSC ERROR: See docs/index.html for manual pages.<br>[5]PETSC ERROR: ------------------------------------------------------------------------<br>
[5]PETSC ERROR: ./main on a linux-opt named node164 by s1459295 Fri Sep 3 17:44:26 2010<br>[5]PETSC ERROR: Libraries linked from /home/s1459295/install/petsc-dev/linux-optimized/lib<br>[5]PETSC ERROR: Configure run at Fri Sep 3 15:28:20 2010<br>
[5]PETSC ERROR: Configure options --with-debugging=0<br>[5]PETSC ERROR: ------------------------------------------------------------------------<br>[5]PETSC ERROR: MatMult() line 1976 in src/mat/interface/matrix.c<br>[5]PETSC ERROR: KSPInitialResidual() line 55 in src/ksp/ksp/interface/itres.c<br>
[5]PETSC ERROR: KSPSolve_GMRES() line 236 in src/ksp/ksp/impls/gmres/gmres.c<br>[5]PETSC ERROR: KSPSolve() line 427 in src/ksp/ksp/interface/itfunc.c<br><br><br><br><br><br><div class="gmail_quote">2010/9/3 Jed Brown <span dir="ltr"><<a href="mailto:jed@59a2.org">jed@59a2.org</a>></span><br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">On Fri, 3 Sep 2010 09:15:09 -0600 (GMT-06:00), Shri <<a href="mailto:abhyshr@mcs.anl.gov">abhyshr@mcs.anl.gov</a>> wrote:<br>
> Leo,<br>
> Jed and I both tried running a MatLoad example (see src/dm/da/examples/tests/ex35.c) with petsc configured in debug mode and had issues running it with your grid size 202 X 102 X 102. This example runs in a reasonable amount of time with the optimized petsc build.<br>
> I suggest running your code on a smaller grid (100 X100 X 100 seems to work fine in the debug mode) for now, get it working and then using the optimized petsc build (--debugging=0) for the large grid size.<br>
<br>
</div>I wrote an improved sort that doesn't have this bad behavior, but I'm<br>
trying to figure out why the old one is okay in optimized mode. See<br>
here:<br>
<br>
<a href="http://gist.github.com/563948" target="_blank">http://gist.github.com/563948</a><br>
<br>
The MEDIAN macro chooses the median of the first, last, and middle<br>
element in the segment. Performance looks like<br>
<br>
FAST -O1: 2.0 seconds<br>
FAST -O0: 5 seconds<br>
SLOW -O1: 2.3 seconds<br>
SLOW -O0: more than an hour<br>
<br>
SLOW, which is basically the old implementation does a lot of shuffling<br>
with the array and ends up hitting a bad case for quicksort (even with<br>
the median choice). What I don't understand, is that apparently<br>
building SLOW with -O1 transforms the algorithm into something<br>
asymptotically faster. The assembly is a few hundred lines, and it's<br>
not obvious to me what this transform would be.<br>
<font color="#888888"><br>
Jed<br>
</font></blockquote></div><br>