[petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code
Matthew Knepley
knepley at gmail.com
Tue May 7 19:17:42 CDT 2024
On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users at mcs.anl.gov> wrote:
> Hello everyone, I hope this email finds you well. My Name is Sawsan
> Shatanawi, and I was developing a Fortran code for simulating groundwater
> flow in a 3D system with nonlinear behavior. I solved the nonlinear system
> using the PCG solver and
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hello everyone,
>
>
>
> I hope this email finds you well.
>
>
>
> My Name is Sawsan Shatanawi, and I was developing a Fortran code for
> simulating groundwater flow in a 3D system with nonlinear behavior. I
> solved the nonlinear system using the PCG solver and Picard iteration, but
> I did not get good results although I checked my matrix and RHS and
> everything, I decided to change my solver to Newton Rapson method.
> I checked PETSc documents but I have a few questions:
> 1) My groundwater system is time-dependent, so should I use TS only
> instead of SNES?
>
You could use TS, but it is not necessary. You could use SNES and your own
timestepping. THe advantage of TS is that you can try many different
timesteppers without recoding (just like you can
try many different linear and nonlinear solvers).
> 2) My system has its deltaT, would using deltaT as dt affect my solver, or
> is it better to use TS-PETSc dt? Also, would using PETSc dt affect the
> simulation of the groundwater system
>
It sounds like your dt comes from your timestepper. If you use TS, you
would use the dt from that.
> 3) I want my Jacobian matrix to be calculated by PETSc automatically
>
PETSc can calculate a full Jacobian for smaller problems, or a
finite-difference Jacobian for any problem (but this impacts the solver).
It should be straightfoward to code up the analytic Jacobian. Is there a
reason it would be a problem?
> 4) Do I need to define and calculate the residual vector?
>
Yes.
> My A-Matrix contains coefficients and external sources and my RHS vector
> includes the boundary conditions
>
It is strange that your matrix would contain source terms. Do they depend
on the flow fields?
Barry is right, you should consider PFlotran, and at least know why it
would not work for your problem if you don't use it.
Thanks,
Matt
>
> Please find the attached file contains a draft of my code
>
> Thank you in advance for your time and help.
>
>
> Best regards,
>
> Sawsan
>
> ------------------------------
> *From:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Sent:* Tuesday, January 16, 2024 10:43 AM
> *To:* Junchao Zhang <junchao.zhang at gmail.com>
> *Cc:* Barry Smith <bsmith at petsc.dev>; Matthew Knepley <knepley at gmail.com>;
> Mark Adams <mfadams at lbl.gov>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> Hello all,
>
> Thank you for your valuable help. I will do your recommendations and hope
> it will run without any issues.
>
> Bests,
> Sawsan
>
> ------------------------------
> *From:* Junchao Zhang <junchao.zhang at gmail.com>
> *Sent:* Friday, January 12, 2024 8:46 AM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* Barry Smith <bsmith at petsc.dev>; Matthew Knepley <knepley at gmail.com>;
> Mark Adams <mfadams at lbl.gov>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> Hi, Sawsan,
> First in test_main.F90, you need to call VecGetArrayF90(temp_solution,
> H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as
> Barry mentioned.
> Secondly, in the loop of test_main.F90, it calls GW_solver(). Within
> it, it calls PetscInitialize()/PetscFinalize(). But without MPI being
> initialized, PetscInitialize()/PetscFinalize()* can only be called once. *
> do timestep =2 , NTSP
> call GW_boundary_conditions(timestep-1)
> !print *,HNEW(1,1,1)
> call GW_elevation()
> ! print *, GWTOP(2,2,2)
> call GW_conductance()
> ! print *, CC(2,2,2)
> call GW_recharge()
> ! print *, B_Rech(5,4)
> call GW_pumping(timestep-1)
> ! print *, B_pump(2,2,2)
> call GW_SW(timestep-1)
> print *,B_RIVER (2,2,2)
> call GW_solver(timestep-1,N)
> call GW_deallocate_loop()
> end do
>
> A solution is to delete PetscInitialize()/PetscFinalize() in
> GW_solver_try.F90 and move it to test_main.F90, outside the do loop.
>
> diff --git a/test_main.F90 b/test_main.F90
> index b5997c55..107bd3ee 100644
> --- a/test_main.F90
> +++ b/test_main.F90
> @@ -1,5 +1,6 @@
> program test_GW
>
> +#include <petsc/finclude/petsc.h>
> use petsc
> use GW_constants
> use GW_param_by_user
> @@ -8,6 +9,9 @@ program test_GW
> implicit none
> integer :: N
> integer :: timestep
> + PetscErrorCode ierr
> +
> + call PetscInitialize(ierr)
> call GW_domain(N)
> !print *, "N=",N
> !print *, DELTAT
> @@ -37,4 +41,5 @@ program test_GW
> end do
> print *, HNEW(NCOL,3,2)
> call GW_deallocate ()
> + call PetscFinalize(ierr)
> end program test_GW
>
> With that, the MPI error will be fixed. The code could run to
> gw_deallocate () before abort. There are other memory errors. You can
> install/use valgrind to fix them. Run it with valgrind ./GW.exe and look
> through the output
>
>
> Thanks.
> --Junchao Zhang
>
>
> On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad <
> sawsan.shatanawi at wsu.edu> wrote:
>
> Hello,
>
> Thank you all for your help.
>
> I have changed VecGetArray to VecGetArrayF90, and the location of destory
> call. but I want to make sure that VecGet ArrayF90 is to make a new array(
> vector) that I can use in the rest of my Fortran code?
>
> when I run it and debugged it, I got
>
> 5.2000000E-03
> 50.00000
> 10.00000
> 0.0000000E+00
> PETSC: Attaching gdb to
> /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065
> on display :0.0 on machine sn16
> Unable to start debugger in xterm: No such file or directory
> 0.0000000E+00
> Attempting to use an MPI routine after finalizing MPICH
> srun: error: sn16: task 0: Exited with exit code 1
> [sawsan.shatanawi at login-p2n02 SS_GWM]$ gdb ./GW/exe
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN-wliID1$
> <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>
> >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN7AsBrfd$
> <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>
> >...
> ./GW/exe: No such file or directory.
> (gdb) run
> Starting program:
> No executable file specified.
> Use the "file" or "exec-file" command.
> (gdb) bt
> No stack.
> (gdb)
>
> If the highlighted line is the error, I don't know why when I write gdb ,
> it does not show me the location of error
> The code : sshatanawi/SS_GWM (github.com)
> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$>
>
>
> I really appreciate your helps
>
> Sawsan
> ------------------------------
> *From:* Barry Smith <bsmith at petsc.dev>
> *Sent:* Wednesday, January 10, 2024 5:35 PM
> *To:* Junchao Zhang <junchao.zhang at gmail.com>
> *Cc:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>; Mark Adams <
> mfadams at lbl.gov>; petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
>
> On Jan 10, 2024, at 6:49 PM, Junchao Zhang <junchao.zhang at gmail.com>
> wrote:
>
> Hi, Sawsan,
> I could build your code and I also could gdb it.
>
> $ gdb ./GW.exe
> ...
> $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault.
> 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0,
> ia=0x7fffffffa75c, ierr=0x0) at
> /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257
> 257 *ierr = VecGetArray(*x, &lx);
> (gdb) bt
> #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0,
> ia=0x7fffffffa75c, ierr=0x0) at
> /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257
> #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at
> GW_solver_try.F90:169
> #2 0x000000000040c6a8 in test_gw () at test_main.F90:35
>
> ierr=0x0 caused the segfault. See
> https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN1gHtGF9$
> <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>,
> you should use VecGetArrayF90 instead.
>
> BTW, Barry, the code
> https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN38ZAbIr$
> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$>
> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find
> petsc Fortran examples doing VecGetArray. Do we still support it?
>
>
> This is not the correct calling sequence for VecGetArray() from
> Fortran.
>
> Regardless, definitely should not be writing any new code that uses
> VecGetArray() from Fortran. Should use VecGetArrayF90().
>
>
> --Junchao Zhang
>
>
> On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users
> <petsc-users at mcs.anl.gov> wrote:
>
> Hello all,
>
> I hope you are doing well.
>
> Generally, I use gdb <the name of my exe.file> to debug the code.
> I got the attached error message.
>
> I have tried to add the flag -start_in_debugger in the make file, but it
> didn't work, so it seems I was doing it in the wrong way
>
> This is the link for the whole code: sshatanawi/SS_GWM (github.com)
> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>
>
> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>
> GitHub - sshatanawi/SS_GWM
> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>
> Contribute to sshatanawi/SS_GWM development by creating an account on
> GitHub.
> github.com
> <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$>
> **
>
> You can read the description of the code in " Model Desprciption.pdf"
> the compiling file is makefile_f90 where you can find the linked code
> files
>
> I really appreciate your help
>
> Bests,
> Sawsan
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Friday, January 5, 2024 4:53 AM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* Matthew Knepley <knepley at gmail.com>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> This is a segv. As Matt said, you need to use a debugger for this or add
> print statements to narrow down the place where this happens.
>
> You will need to learn how to use debuggers to do your project so you
> might as well start now.
>
> If you have a machine with a GUI debugger that is easier but command line
> debuggers are good to learn anyway.
>
> I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...)
> and use a GUI debugger (eg, Totalview or DDT) if available.
>
> Mark
>
>
> On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via
> petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Hello Matthew,
>
> Thank you for your help. I am sorry that I keep coming back with my error
> messages, but I reached a point that I don't know how to fix them, and I
> don't understand them easily.
> The list of errors is getting shorter, now I am getting the attached error
> messages
>
> Thank you again,
>
> Sawsan
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Wednesday, December 20, 2023 6:54 PM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* Barry Smith <bsmith at petsc.dev>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users
> <petsc-users at mcs.anl.gov> wrote:
>
> Hello Barry,
>
> Thank you a lot for your help, Now I am getting the attached error message.
>
>
> Do not destroy the PC from KSPGetPC()
>
> THanks,
>
> Matt
>
>
> Bests,
> Sawsan
> ------------------------------
> *From:* Barry Smith <bsmith at petsc.dev>
> *Sent:* Wednesday, December 20, 2023 6:32 PM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* Mark Adams <mfadams at lbl.gov>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
> Instead of
>
> call PCCreate(PETSC_COMM_WORLD, pc, ierr)
> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
> call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the
> KSP solver
>
> do
>
> call KSPGetPC(ksp,pc,ierr)
> call PCSetType(pc, PCILU,ierr)
>
> Do not call KSPSetUp(). It will be taken care of automatically during the
> solve
>
>
>
> On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hello,
> I don't think that I set preallocation values when I created the matrix,
> would you please have look at my code. It is just the petsc related part
> from my code.
> I was able to fix some of the error messages. Now I have a new set of
> error messages related to the KSP solver (attached)
>
> I appreciate your help
>
> Sawsan
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Wednesday, December 20, 2023 6:44 AM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> Did you set preallocation values when you created the matrix?
> Don't do that.
>
> On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <
> sawsan.shatanawi at wsu.edu> wrote:
>
> Hello,
>
> I am trying to create a sparse matrix( which is as I believe a zero
> matrix) then adding some nonzero elements to it over a loop, then
> assembling it
>
> Get Outlook for iOS
> <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$>
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Wednesday, December 20, 2023 2:48 AM
> *To:* Shatanawi, Sawsan Muhammad <sawsan.shatanawi at wsu.edu>
> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> I am guessing that you are creating a matrix, adding to it, finalizing it
> ("assembly"), and then adding to it again, which is fine, but you are
> adding new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
>
> Mark
>
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users
> <petsc-users at mcs.anl.gov> wrote:
>
> Hello everyone,
>
> I hope this email finds you well.
>
> My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>
> Thank you in advance for your time and assistance.
>
> Best regards,
>
> Sawsan
>
> <Matrix_RHS.F90><out.txt><solver.F90>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN8rPXZEB$
> <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>
>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjN8rPXZEB$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a5WSXOX8ewfPLoPMZrwcIgaFqi8MKa016aCtKSJ_a6Ic0bWJcexX38TUj1Whfm88oF2dghzS5-WjNzGgO5Ai$ >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240507/00a1cd29/attachment-0001.html>
More information about the petsc-users
mailing list