<html><head><base href="x-msg://567/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div> Most of the time is in the MatLoad_MPIAIJ() (you are running in parallel right) so you could compile and run with gprof to see where in the load is most of the time. It is probably in disk reads and MPI sends.<div>But the repartioning for the DA is not taking much of the time.</div><div><br></div><div> Barry</div><div><br><div><div>On Sep 2, 2010, at 12:46 PM, Shri wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="font-family: 'Times New Roman'; font-size: 16px; "><div><br class="Apple-interchange-newline">> That code cannot be right. When you copy the App to A it does not have the same nonzero pattern. You should not use a MatCopy() there you should use a MatHeaderMerge() or a MatHeaderReplace() whichever one is right.</div><br><div>Thanks,i was searching for a similar function but did not find it hence used MatCopy. I've fixed this and pushed it.</div><br><div> > Then try running with a big DA to see how long it takes.</div><div> </div><div>I tried MatLoad on Leo's 3D grid (202 X 102 X 102) but it takes forever to load. Hence, I tried it on a relatively smaller grid (100 X 100 X 100) and here are the log summary numbers that i got.</div><br><div><div>Event Count Time (sec) Flops --- Global --- --- Stage --- Total</div><div> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div><div>------------------------------------------------------------------------------------------------------------------------</div><br><div>--- Event Stage 0: Main Stage</div><br><div>MatAssemblyBegin 4 1.0 2.6508e-0134.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 3 0 0 0 5 3 0 0 0 6 0</div><div>MatAssemblyEnd 4 1.0 4.2604e-01 1.0 0.00e+00 0.0 1.2e+01 2.0e+04 3.6e+01 9 0 63 1 28 9 0 63 1 38 0</div><div>MatGetSubMatrice 1 1.0 5.9222e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 13 0 0 0 4 13 0 0 0 5 0</div><div>MatLoad 1 1.0 3.2231e+00 1.0 0.00e+00 0.0 1.1e+01 4.0e+06 4.9e+01 70 0 58100 38 70 0 58100 52 0</div><div>------------------------------------------------------------------------------------------------------------------------</div></div><br><br><div>Shri</div><div> </div><div> Barry</div><div><br><div><div>On Sep 2, 2010, at 10:17 AM, Leo van Kampenhout wrote:</div><br><blockquote>Hello Shri,<br><br>thanks for the fix. The error is now gone but the loading of the Mat takes now infinitely long.. help!!<br><br>Leo<br><br><br><div class="gmail_quote">2010/9/2 Shri<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:abhyshr@mcs.anl.gov" target="_blank">abhyshr@mcs.anl.gov</a>></span><br><br>><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; "><div>Pushed. Please let me know if you get any errors.<br><div><font color="#888888">Shri</font><div><br>><div class="h5"><br><br>----- Barry Smith wrote:<br>><div> if (size > 1) { </div><div> /* change viewer to display matrix in natural ordering */</div><div> ierr = MatShellSetOperation(A, MATOP_VIEW, (void (*)(void)) MatView_MPI_DA);CHKERRQ(ierr);</div><br>><div> /* turn off loading of matrix because loading would require proper permutation I don't feel like writing now */</div><div> ierr = MatShellSetOperation(A, MATOP_LOAD, (void (*)(void)) 0);CHKERRQ(ierr);</div><br>><div> }</div><br><br><div> Shri,</div><br><div> Could you please add the support for this? And push to petsc-dev</div><br><div> Thanks</div><br><div> Barry</div><br><br><div><div>On Sep 1, 2010, at 10:23 AM, Leo van Kampenhout wrote:</div><br>><br><blockquote><br>Using a DA, of equal size and type as the other one. The curious thing is that there are no problems on a single core run. (but that is not what I want ;)<br><br><br><br><div class="gmail_quote">2010/9/1 Barry Smith<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span><br><br>><br>><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; "><div style="word-wrap: break-word; "><br> Did you get the matrix in the writer code using a DA or some other code that puts the vector from the 2d or 3d problem in the natural ordering on the binary file.<div><br>><br>><br></div><font color="#888888"><div> Barry</div></font><div><div><div><br><div><div>On Sep 1, 2010, at 9:55 AM, Leo van Kampenhout wrote:</div><br><blockquote>Hi all,<span class="Apple-converted-space"> </span><br><br>I'm having trouble with MatLoad on Petsc-Dev. Not sure if this problem also exists in the main PETSc 3.1 since my code won't compile on that.<span class="Apple-converted-space"> </span><br><br>><br>> The problematic calls are as follows:<span class="Apple-converted-space"> </span><br><br><span style="font-family: 'courier new', monospace; "> call PetscViewerBinaryOpen(PETSC_COMM_WORLD,file2,</span><br><br>><span style="font-family: 'courier new', monospace; "> & FILE_MODE_READ,fd,ierr)</span><br><br>><span style="font-family: 'courier new', monospace; "> call DAGetMatrix(da,MATMPIAIJ,A,ierr)</span><br><br>><span class="Apple-converted-space"> </span><br>><span style="font-family: 'courier new', monospace; "> <span class="Apple-converted-space"> </span></span><span style="font-family: 'courier new', monospace; ">call MatLoad(A,fd,ierr)</span><br><br>><br>which supposedly will load a matrix A which is stored by the following commands:<br><br>><span class="Apple-converted-space"> </span><br>><br><span style="font-family: 'courier new', monospace; "> call PetscViewerBinaryOpen(PETSC_COMM_WORLD,file2,</span><br><br>><span style="font-family: 'courier new', monospace; "> & FILE_MODE_WRITE,fd,ierr)</span><br><br>><span class="Apple-converted-space"> </span><br>><span style="font-family: 'courier new', monospace; "> call MatView(A,fd,ierr)</span><br><br>by another program (on a single core). The MatLoad() does work perfectly on a single core, however two or more cores make the program crash with the following error (see below)<br><br>><br>><span class="Apple-converted-space"> </span><br>> The programcall is<span class="Apple-converted-space"> </span><br><div style="margin-left: 40px; "><span style="font-family: 'courier new', monospace; "> mpirun -n 2 ./main</span><br></div><br>Is something broken with Petsc or do I need to set some extra parameters?<span class="Apple-converted-space"> </span><br><br>><br>><span class="Apple-converted-space"> </span><br>> Thanks in advance,<span class="Apple-converted-space"> </span><br><br>Leo van Kampenhout<br><br>PS. I'm using a build from last friday, since when I updated petsc-dev today using Mercurial I couldnt get it to configure or compile :S (which is another subject)<br><br>><br>><span class="Apple-converted-space"> </span><br>><br><br>[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>[0]PETSC ERROR: No support for this operation for this object type!<br>[0]PETSC ERROR: MatLoad is not supported for type: mpiaij!<br><br>><br>><span class="Apple-converted-space"> </span><br>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: Petsc Development HG revision: 84fec3ca75412473b06251029b58f2a68f1334f3 HG Date: Fri Aug 27 07:06:08 2010 +0200<br><br>><br>><span class="Apple-converted-space"> </span><br>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[0]PETSC ERROR: See docs/index.html for manual pages.<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br><br>><br>><span class="Apple-converted-space"> </span><br>> [0]PETSC ERROR: ./main on a arch-linu named wingtip72 by csg4035 Wed Sep 1 15:52:11 2010<br>[0]PETSC ERROR: Libraries linked from /net/users/csg/csg4035/install/petsc-dev/arch-linux-gnu-c-debug/lib<br>[0]PETSC ERROR: Configure run at Fri Aug 27 08:37:32 2010<br><br>><br>><span class="Apple-converted-space"> </span><br>> [0]PETSC ERROR: Configure options<span class="Apple-converted-space"> </span><br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: MatLoad() line 843 in src/mat/interface/matrix.c<br><br><br><br>><br>></blockquote></div><br></div></div></div></div></blockquote></div><br><br>></blockquote></div><br><br></div></div></div></div></blockquote></div><br><br>></blockquote></div><br></div><br></span></span><br class="Apple-interchange-newline"></blockquote></div><br></div></body></html>