[petsc-users] Customizing NASM subsnes

[Takahashi, Tadanaga]

Thank you. I am now able to pull each subsnes, change its snes type through
the API, and set a prefix. This is my updated code:

   SNES           snes, subsnes;
   PetscMPIInt    rank, size;
   ...
   ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);
   ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);
   ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);
   ...
   ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
   ierr = SNESSetUp(snes); CHKERRQ(ierr);
   PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);
   PetscBarrier(NULL);
   for (i=0; i<size; i++) {
      char prefix[10];
      sprintf(prefix,"sub_%d_",i);
      if(i==rank) {
         ierr = SNESNASMGetNumber(snes,&Nd);
         printf("rank = %d has %d block(s)\n",i,Nd);
         if (i <size-1) {
            SNESNASMGetSNES(snes,0,&subsnes);
            SNESSetType(subsnes,SNESNEWTONLS); CHKERRQ(ierr); // newton for
regular domains
         } else {
            SNESNASMGetSNES(snes,0,&subsnes);
            SNESSetType(subsnes,SNESFAS); CHKERRQ(ierr); // fas for last
domain
         }
         SNESSetOptionsPrefix(subsnes,prefix);
      }
   }
   ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
   ...
   ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);

However, I still cannot change SNES, KSP, and PC types for
individual domains through the command arguments. I checked the subdomains
with -snes_view ::ascii_info_detail and it does show that the prefixes are
properly changed. It also shows that the SNES type for the last domain was
successfully changed. But for some reason, I only have access to the SNES
viewer options during runtime. For example, if I run mpiexec -n 4 ./test1
-sub_0_ksp_type gmres -help | grep sub_0 I get the output:

Viewer (-sub_0_snes_convergence_estimate) options:
  -sub_0_snes_convergence_estimate ascii[:[filename][:[format][:append]]]:
Prints object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_convergence_estimate binary[:[filename][:[format][:append]]]:
Saves object to a binary file (PetscOptionsGetViewer)
  -sub_0_snes_convergence_estimate draw[:[drawtype][:filename|format]]
Draws object (PetscOptionsGetViewer)
  -sub_0_snes_convergence_estimate socket[:port]: Pushes object to a Unix
socket (PetscOptionsGetViewer)
  -sub_0_snes_convergence_estimate saws[:communicatorname]: Publishes
object to SAWs (PetscOptionsGetViewer)
Viewer (-sub_0_snes_view_pre) options:
  -sub_0_snes_view_pre ascii[:[filename][:[format][:append]]]: Prints
object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_view_pre binary[:[filename][:[format][:append]]]: Saves
object to a binary file (PetscOptionsGetViewer)
  -sub_0_snes_view_pre draw[:[drawtype][:filename|format]] Draws object
(PetscOptionsGetViewer)
  -sub_0_snes_view_pre socket[:port]: Pushes object to a Unix socket
(PetscOptionsGetViewer)
  -sub_0_snes_view_pre saws[:communicatorname]: Publishes object to SAWs
(PetscOptionsGetViewer)
Viewer (-sub_0_snes_test_jacobian_view) options:
  -sub_0_snes_test_jacobian_view ascii[:[filename][:[format][:append]]]:
Prints object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_view binary[:[filename][:[format][:append]]]:
Saves object to a binary file (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_view draw[:[drawtype][:filename|format]] Draws
object (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_view socket[:port]: Pushes object to a Unix
socket (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_view saws[:communicatorname]: Publishes object
to SAWs (PetscOptionsGetViewer)
Viewer (-sub_0_snes_test_jacobian_display) options:
  -sub_0_snes_test_jacobian_display ascii[:[filename][:[format][:append]]]:
Prints object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_display
binary[:[filename][:[format][:append]]]: Saves object to a binary file
(PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_display draw[:[drawtype][:filename|format]]
Draws object (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_display socket[:port]: Pushes object to a Unix
socket (PetscOptionsGetViewer)
  -sub_0_snes_test_jacobian_display saws[:communicatorname]: Publishes
object to SAWs (PetscOptionsGetViewer)
Viewer (-sub_0_ksp_converged_reason) options:
  -sub_0_ksp_converged_reason ascii[:[filename][:[format][:append]]]:
Prints object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_ksp_converged_reason binary[:[filename][:[format][:append]]]:
Saves object to a binary file (PetscOptionsGetViewer)
  -sub_0_ksp_converged_reason draw[:[drawtype][:filename|format]] Draws
object (PetscOptionsGetViewer)
  -sub_0_ksp_converged_reason socket[:port]: Pushes object to a Unix socket
(PetscOptionsGetViewer)
  -sub_0_ksp_converged_reason saws[:communicatorname]: Publishes object to
SAWs (PetscOptionsGetViewer)
Viewer (-sub_0_snes_converged_reason) options:
  -sub_0_snes_converged_reason ascii[:[filename][:[format][:append]]]:
Prints object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_converged_reason binary[:[filename][:[format][:append]]]:
Saves object to a binary file (PetscOptionsGetViewer)
  -sub_0_snes_converged_reason draw[:[drawtype][:filename|format]] Draws
object (PetscOptionsGetViewer)
  -sub_0_snes_converged_reason socket[:port]: Pushes object to a Unix
socket (PetscOptionsGetViewer)
  -sub_0_snes_converged_reason saws[:communicatorname]: Publishes object to
SAWs (PetscOptionsGetViewer)
Viewer (-sub_0_snes_view) options:
  -sub_0_snes_view ascii[:[filename][:[format][:append]]]: Prints object to
stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_view binary[:[filename][:[format][:append]]]: Saves object to
a binary file (PetscOptionsGetViewer)
  -sub_0_snes_view draw[:[drawtype][:filename|format]] Draws object
(PetscOptionsGetViewer)
  -sub_0_snes_view socket[:port]: Pushes object to a Unix socket
(PetscOptionsGetViewer)
  -sub_0_snes_view saws[:communicatorname]: Publishes object to SAWs
(PetscOptionsGetViewer)
Viewer (-sub_0_snes_view_solution) options:
  -sub_0_snes_view_solution ascii[:[filename][:[format][:append]]]: Prints
object to stdout or ASCII file (PetscOptionsGetViewer)
  -sub_0_snes_view_solution binary[:[filename][:[format][:append]]]: Saves
object to a binary file (PetscOptionsGetViewer)
  -sub_0_snes_view_solution draw[:[drawtype][:filename|format]] Draws
object (PetscOptionsGetViewer)
  -sub_0_snes_view_solution socket[:port]: Pushes object to a Unix socket
(PetscOptionsGetViewer)
  -sub_0_snes_view_solution saws[:communicatorname]: Publishes object to
SAWs (PetscOptionsGetViewer)
Option left: name:-sub_0_ksp_type value: gmres


Do you know what could be causing this?

On Fri, Jun 17, 2022 at 10:00 AM Barry Smith <bsmith at petsc.dev> wrote:

>
>      MPI_Comm_size(PETSC_COMM_WORLD,&size);
>      MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
>
> SNESNASMGetSNES(snes,0,&subsnes);
>>  char prefix[10];
>>  sprintf(prefix,"sub_%d_",rank);
>>  SNESSetOptionsPrefix(subsnes,prefix);
>>
>
>
>
> On Jun 17, 2022, at 9:35 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Jun 17, 2022 at 9:22 AM Takahashi, Tadanaga <tt73 at njit.edu> wrote:
>
>> I'm having some trouble pulling out the subsolver. I tried to use
>> SNESNASMGetSNES in a loop over each subdomain. However I get an error when
>> I run the code with more than one MPI processors. Here is a snippet from my
>> code:
>>
>>    SNES           snes, subsnes;
>>    PetscMPIInt    rank, size;
>>    ...
>>    ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);
>>    ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);
>>    ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);
>>    ...
>>    ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
>>    ierr = SNESSetUp(snes); CHKERRQ(ierr);
>>    PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);
>>    for (i=0; i<size; i++) {
>>       PetscPrintf(PETSC_COMM_WORLD, "rank = %d\n",i);
>>       SNESNASMGetSNES(snes,i,&subsnes);
>>       // char prefix[10];
>>       // sprintf(prefix,"sub_%d_",i);
>>       // SNESSetOptionsPrefix(subsnes,prefix);
>>    }
>>    ...
>>    ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);
>>
>>
>> And, here is the output of the code when I run with 2 MPI procs:
>>
>
> SNESNASMGetSNES() gets the local subsolvers. It seems you only have one
> per process.
> You can check
> https://petsc.org/main/docs/manualpages/SNES/SNESNASMGetNumber/
>
> Notice that your current code will not work because, according to your
> explanation, you only want to change
> the prefix on a single rank, so you need to check the rank when you do it.
>
>   Thanks,
>
>      Matt
>
>
>> takahashi at ubuntu:~/Desktop/MA-DDM/C/Rectangle$ mpiexec -n 2 ./test1
>> Size = 2
>> rank = 0
>> rank = 1
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: No such subsolver
>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.17.1, unknown
>> [0]PETSC ERROR: ./test1 on a linux-gnu-c-debug named ubuntu by takahashi
>> Fri Jun 17 06:06:38 2022
>> [0]PETSC ERROR: Configure options --with-mpi-dir=/usr --with-fc=0
>> [0]PETSC ERROR: #1 SNESNASMGetSNES() at
>> /home/takahashi/Desktop/petsc/src/snes/impls/nasm/nasm.c:923
>>
>>
>> ===================================================================================
>> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> =   RANK 0 PID 976566 RUNNING AT ubuntu
>> =   KILLED BY SIGNAL: 9 (Killed)
>>
>> ===================================================================================
>>
>> This error doesn't occur when I run this without MPI. However, I tried to
>> change the prefix of the subdomain to `sub_0_` but I am not able to change
>> the snes_type using this prefix. Running ./test1 -snes_view -help | grep
>> sub_0_snes_type prints nothing.
>>
>> On Thu, Jun 16, 2022 at 6:23 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jun 16, 2022 at 5:57 PM tt73 <tt73 at njit.edu> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I am using  NASM as the outer solver for a nonlinear problem. For one
>>>> of the subdomains, I want to run the local solve with a different set of
>>>> options form the others. Is there any way to set options for each
>>>> subdomain?
>>>>
>>>
>>> I can see two ways:
>>>
>>>   1) Pull out the subsolver and set it using the API
>>>
>>>   2) Pull out the subsolver and give it a different prefix
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220617/c55d5af1/attachment.html>