suggestions for debugging code

[Randall Mackie]

I have run into a very difficult debugging problem. I have recently made some
modifications to my PETSc code, to add some new features. When I compiled the
code in debug mode (we are using the Intel compilers and mvapich on Infiniband),
the code runs fine with any number of processes.

When the code is compiled in optimize mode, it runs fine on, say, up to 32 processes,
but not 64, bombing out someplace strange, with a Segmentation Violation.

I've tried using Valgrind, but you can't use it with PETSc and my code compiled in
Debug mode because the code finishes successfully, and the other problem I have with
Valgrind + mvapich is there are about a million messages spewed out, making it
extremely difficult to see if there are really any issues in MY code. I've thought
to have PETSc download and compile MPICH2, which I would hope would produce less
output from Valgrind.

Anyone have any suggestions on how to debug this tricky situation? Any suggestions
would be greatly appreciated.

Randy