[petsc-users] Problem with solving Poisson eqn for some cases

Smith, Barry F. bsmith at mcs.anl.gov
Mon Mar 26 15:55:10 CDT 2018 

   Run in the debugger and step/next through the code to see what code path it is taking and why it doesn't seem to follow the intended code path.



> On Mar 26, 2018, at 7:10 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Mon, Mar 26, 2018 at 1:15 AM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> On 24/3/2018 6:00 AM, Matthew Knepley wrote:
>> On Fri, Mar 23, 2018 at 5:23 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>> 
>> On 21/3/2018 7:47 PM, Matthew Knepley wrote:
>>> On Wed, Mar 21, 2018 at 4:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>> 
>>> On 19/3/2018 6:32 PM, Matthew Knepley wrote:
>>>> On Mon, Mar 19, 2018 at 5:19 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>> 
>>>> On 17/3/2018 1:15 AM, Matthew Knepley wrote:
>>>>> On Fri, Mar 16, 2018 at 12:54 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>> 
>>>>> On 15/3/2018 6:21 PM, Matthew Knepley wrote:
>>>>>> On Thu, Mar 15, 2018 at 3:51 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I'm running a CFD code which solves the momentum and Poisson eqns.
>>>>>> 
>>>>>> Due to poor scaling with HYPRE at higher cpu no., I decided to try using PETSc with boomeramg and gamg.
>>>>>> 
>>>>>> I tested for some small cases and it work well. However, for the large problem which has poor scaling, it gives an error when I change my Poisson solver from pure HYPRE to PETSc with boomeramg and gamg.
>>>>>> 
>>>>>> The error is :
>>>>>> 
>>>>>> Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>> 
>>>>>> I tried using:
>>>>>> 
>>>>>> -poisson_ksp_type richardson -poisson_pc_type hypre -poisson_pc_type_hypre boomeramg
>>>>>> 
>>>>>> -poisson_ksp_type gmres -poisson_pc_type hypre -poisson_pc_type_hypre boomeramg
>>>>>> 
>>>>>> -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1
>>>>>> 
>>>>>> but they all gave similar error.
>>>>>> 
>>>>>> So why is this so? How should I troubleshoot? I am now running a debug ver of PETSc to check the error msg.
>>>>>> 
>>>>>> 1) For anything like this, we would like to see a stack trace from the debugger or valgrind output.
>>>>>> 
>>>>>> 2) We do have several Poisson examples. Does it fail for you on those?
>>>>> Hi,
>>>>> 
>>>>> Can you recommend me some suitable egs? Esp in Fortran?
>>>>> 
>>>>> Here is 2D Poisson
>>>>> 
>>>>>   https://bitbucket.org/petsc/petsc/src/4b6141395f14f0c7d1415a2ff0158eec75a27d63/src/snes/examples/tutorials/ex5f.F90?at=master&fileviewer=file-view-default
>>>>>  
>>>>>> 
>>> Hi,
>>> 
>>> I tried to use the different options like ML, MG BoomerAMG and while they worked for the small problem, it failed for the large problem. I checked and compared the 2D Poisson example with my 3D Poisson subroutine. I found that my subroutine is very similar to my momentum subroutine, which is based on KSPSolve:
>>> 
>>> 0. DMDACreate, DMDACreate3d etc
>>> 1. Assemble matrix
>>> 2. KSPSetOperators, KSPSetType etc
>>> 3. KSPSolve
>>> 
>>> However, the 2D Poisson example uses SNESSetDM, SNESSolve etc. So does it matter if I use SNESSolve or KSPSolve?
>>> 
>>> Not if you are using Algebraic Multigrid. "It failed for the large problem" is not descriptive enough. Please send the output of
>>> 
>>>   -ksp_view -ksp_monitor_true_residual
>>>  
>>> I also didn't set any nullspace. Is this required for a Poisson eqn solve?
>>> 
>>> If you only have the constant null space, no. Most AMG packages do that one by default.
>>> 
>>>   Matt
>> Hi,
>> 
>> I tried to use 
>> 
>> 1. -ksp_view -ksp_monitor_true_residual
>> 2. -poisson_ksp_view -poisson_ksp_monitor_true_residual
>> 
>> However, this did not give any output. May I know why?
>> 
>> You need to call
>> 
>>   KSPSetFromOptions()
> Hi Matt, 
> 
> But I thought I already called:
> 
> call KSPSetOperators(ksp,A_mat,A_mat,ierr)
> 
>     call KSPGetPC(ksp,pc,ierr)  
>     
>     call KSPSetOptionsPrefix(ksp,"poisson_",ierr)
> 
>     call KSPSetType(ksp,ksptype,ierr)
> 
>     ksptype=KSPRICHARDSON 
>     
>     call KSPSetType(ksp,ksptype,ierr)
> 
>     call PCSetType(pc,'hypre',ierr)
> 
>     call KSPSetFromOptions(ksp,ierr)
> 
> ...
> 
> Is my sequence of calling wrong?
> 
> If its not viewing, something is wrong. Here is the easiest way to make things work.
> Take an example, and make it do what you want, like view. Then change it until it
> is your own thing.
> 
> The code snippet is right, so there must be something around it that is messing things up.
> 
>   Thanks,
> 
>     Matt
>  
> Thanks.
>> 
>> after you create it. When you launch valgrind you need
>> 
>>   valgrind --trace-children=yes <exec>
>> 
>>    Matt
>>  
>> I also tried valgrind:
>> 
>> ==32157== Memcheck, a memory error detector
>> ==32157== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
>> ==32157== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
>> ==32157== Command: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpirun ./a.out -poisson_ksp_type richardson -poisson_pc_type hypre -poisson_pc_type_hypre boomeramg
>> ==32157==
>> ==32159==
>> ==32159== HEAP SUMMARY:
>> ==32159==     in use at exit: 49,981 bytes in 1,137 blocks
>> ==32159==   total heap usage: 1,542 allocs, 405 frees, 72,030 bytes allocated
>> ==32159==
>> ==32159== LEAK SUMMARY:
>> ==32159==    definitely lost: 0 bytes in 0 blocks
>> ==32159==    indirectly lost: 0 bytes in 0 blocks
>> ==32159==      possibly lost: 0 bytes in 0 blocks
>> ==32159==    still reachable: 49,981 bytes in 1,137 blocks
>> ==32159==         suppressed: 0 bytes in 0 blocks
>> ==32159== Reachable blocks (those to which a pointer was found) are not shown.
>> ==32159== To see them, rerun with: --leak-check=full --show-reachable=yes
>> ==32159==
>> ==32159== For counts of detected and suppressed errors, rerun with: -v
>> ==32159== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6)
>> [362]PETSC ERROR: ------------------------------------------------------------------------
>> [362]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [362]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [362]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [362]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [362]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
>> [362]PETSC ERROR: to get more information on the crash.
>> [362]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [362]PETSC ERROR: Signal received
>> [362]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [362]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017
>> [362]PETSC ERROR: ./a.out on a petsc-3.8.3_intel_rel named std1055 by tsltaywb Thu Mar 22 17:43:54 2018
>> [362]PETSC ERROR: Configure options --with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64 --with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/source/git.hypre.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.8.3_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2" --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2"
>> [362]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 362
>> ==32409==
>> ==32409== HEAP SUMMARY:
>> ==32409==     in use at exit: 52,168 bytes in 1,223 blocks
>> ==32409==   total heap usage: 3,055 allocs, 1,832 frees, 122,188 bytes allocated
>> ==32409==
>> ==32409== 10 bytes in 1 blocks are definitely lost in loss record 89 of 319
>> ==32409==    at 0x4C28A2E: malloc (vg_replace_malloc.c:270)
>> ==32409==    by 0x466742: xmalloc (in /bin/bash)
>> ==32409==    by 0x42EC78: ??? (in /bin/bash)
>> ==32409==    by 0x430462: execute_command_internal (in /bin/bash)
>> ==32409==    by 0x43320D: ??? (in /bin/bash)
>> ==32409==    by 0x433629: ??? (in /bin/bash)
>> ==32409==    by 0x4303BC: execute_command_internal (in /bin/bash)
>> ==32409==    by 0x43110D: execute_command (in /bin/bash)
>> ==32409==    by 0x41D6D5: reader_loop (in /bin/bash)
>> ==32409==    by 0x41CEBB: main (in /bin/bash)
>> ==32409==
>> ==32409== LEAK SUMMARY:
>> ==32409==    definitely lost: 10 bytes in 1 blocks
>> ==32409==    indirectly lost: 0 bytes in 0 blocks
>> ==32409==      possibly lost: 0 bytes in 0 blocks
>> ==32409==    still reachable: 52,158 bytes in 1,222 blocks
>> ==32409==         suppressed: 0 bytes in 0 blocks
>> ==32409== Reachable blocks (those to which a pointer was found) are not shown.
>> ==32409== To see them, rerun with: --leak-check=full --show-reachable=yes
>> ==32409== Reachable blocks (those to which a pointer was found) are not shown.
>> ==32409== To see them, rerun with: --leak-check=full --show-reachable=yes
>> ==32409==
>> ==32409== For counts of detected and suppressed errors, rerun with: -v
>> ==32409== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 8 from 6)
>> ==32157==
>> ==32157== HEAP SUMMARY:
>> ==32157==     in use at exit: 55,861 bytes in 1,202 blocks
>> ==32157==   total heap usage: 3,249 allocs, 2,047 frees, 125,857 bytes allocated
>> ==32157==
>> ==32157== LEAK SUMMARY:
>> ==32157==    definitely lost: 0 bytes in 0 blocks
>> ==32157==    indirectly lost: 0 bytes in 0 blocks
>> ==32157==      possibly lost: 0 bytes in 0 blocks
>> ==32157==    still reachable: 55,861 bytes in 1,202 blocks
>> ==32157==         suppressed: 0 bytes in 0 blocks
>> ==32157== Reachable blocks (those to which a pointer was found) are not shown.
>> ==32157== To see them, rerun with: --leak-check=full --show-reachable=yes
>> ==32157==
>> ==32157== For counts of detected and suppressed errors, rerun with: -v
>> ==32157== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6)
>> 
>> Is this the output I'm supposed to get?
>> 
>> Thanks!
>> 
>> 
>>>  
>>> Thanks
>>>>>> 3) You can also try ML, which is the same type of MG as GAMG. (--download-ml).
>>>>>> 
>>>> I have recompiled PETSc with ML. Is there an example command line options which I can use for ML?
>>>> 
>>>> -pc_type ml
>>>>  
>>>> Another question is generally speaking, is geometric multigrid (GMG) faster than algebraic?
>>>> 
>>>> No, only for the setup time.
>>>>  
>>>> I tested on a small problem and the time taken varies from 1.15min (HYPRE, geometric) to 3.25 (GAMG). BoomerAMG is 1.45min.
>>>> 
>>>> I am not sure what you are running when you say Hypre geometric.
>>>>  
>>>> Besides HYPRE, is there any other GMG I can use? 
>>>> 
>>>> As I said above, it does not converge any faster and the solve is not faster, its all setup time, so small problems will look faster.
>>>> You should be doing 10-20 iterates. If you are doing more, MG is not working.
>>>> 
>>>> If you truly have a structured grid, then use DMDA in PETSc and you can use GMG with
>>>> 
>>>>   -pc_type mg -pc_mg_nlevels <n>
>>>> 
>>>>    Matt
>>>>> My cluster can't connect to the internet. Where can I 1st download it?
>>>>> 
>>>>> Similarly, how can I find out the location of the ext software by myself?
>>>>> 
>>>>> The locations are all in the configure Python modules:
>>>>> 
>>>>>   https://bitbucket.org/petsc/petsc/src/4b6141395f14f0c7d1415a2ff0158eec75a27d63/config/BuildSystem/config/packages/ml.py?at=master&fileviewer=file-view-default
>>>>> 
>>>>>   Thanks,
>>>>> 
>>>>>     Matt
>>>>>  
>>>>>>   Thanks,
>>>>>> 
>>>>>>      Matt
>>>>>>  
>>>>>> 
>>>>>> -- 
>>>>>> Thank you very much.
>>>>>> 
>>>>>> Yours sincerely,
>>>>>> 
>>>>>> ================================================
>>>>>> TAY Wee-Beng (Zheng Weiming) 郑伟明
>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website
>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>>>>>> linkedin: www.linkedin.com/in/tay-weebeng
>>>>>> ================================================
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>> 
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their                                                           experiments lead.
>>>>> -- Norbert Wiener
>>>>> 
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>> 
>>>> https://www.cse.buffalo.edu/~knepley/
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

More information about the petsc-users mailing list