<div dir="ltr"><div dir="ltr">On Fri, Jun 17, 2022 at 11:02 AM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div> You do not need the loop over size. Each rank sets options and options prefix for its local objects and never anyone elses. <div><br></div><div><blockquote type="cite"><div dir="ltr"><div><div><font face="monospace"> </font><span style="font-family:monospace">char prefix[10];</span></div></div></div></blockquote><blockquote type="cite"><div dir="ltr"><div><div><font face="monospace"> sprintf(prefix,"sub_%d_",rank);</font></div></div></div></blockquote><blockquote type="cite"><div dir="ltr"><div><div><span style="font-family:monospace"> SNESNASMGetSNES(snes,0,&subsnes);</span></div></div></div></blockquote><blockquote type="cite"><div dir="ltr"><div><div><font face="monospace"> if (rank <size-1) {<br> SNESSetType(subsnes,SNESNEWTONLS); CHKERRQ(ierr); // newton for regular domains<br> } else {<br> SNESSetType(subsnes,SNESFAS); CHKERRQ(ierr); // fas for last domain <br> }<br> SNESSetOptionsPrefix(subsnes,prefix);<br> }<br> }</font></div></div></div></blockquote><div><br></div><div> To get the prefix to work try calling <span style="font-family:monospace">SNESSetFromOptions(subsnes); immediately after your </span><span style="font-family:monospace">SNESSetOptionsPrefix(subsnes,prefix); call</span></div><div><br></div> Matt, it looks like there may be a bug in NASM, except in one particular case, it never calls SNESSetFromOptions() on the subsenses.</div></div></blockquote><div><br></div><div>I bet Barry is correct. I can fix it, but unfortunately, I leave for a week long conference tomorrow, of which I am an organizer, so I don't</div><div>think I can do it for a week.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Jun 17, 2022, at 10:47 AM, Takahashi, Tadanaga <<a href="mailto:tt73@njit.edu" target="_blank">tt73@njit.edu</a>> wrote:</div><br><div><div dir="ltr"><div><font face="arial, sans-serif">Thank you. I am now able to pull each subsnes, change its snes type through the API, and set a prefix. </font><span style="font-family:arial,sans-serif">This is my updated code: </span></div><div><font face="monospace"><br></font></div><div><font face="monospace"> SNES snes, subsnes;<br> PetscMPIInt rank, size;<br></font></div><div><font face="monospace"> ... </font></div><div><div><font face="monospace"> ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);<br> ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);<br> ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);<br> ... <br> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);<br> ierr = SNESSetUp(snes); CHKERRQ(ierr);<br> PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);<br> PetscBarrier(NULL);<br> for (i=0; i<size; i++) {<br> char prefix[10];<br> sprintf(prefix,"sub_%d_",i);<br> if(i==rank) {<br> ierr = SNESNASMGetNumber(snes,&Nd);<br> printf("rank = %d has %d block(s)\n",i,Nd);<br> if (i <size-1) {<br> SNESNASMGetSNES(snes,0,&subsnes);<br> SNESSetType(subsnes,SNESNEWTONLS); CHKERRQ(ierr); // newton for regular domains<br> } else {<br> SNESNASMGetSNES(snes,0,&subsnes);<br> SNESSetType(subsnes,SNESFAS); CHKERRQ(ierr); // fas for last domain <br> }<br> SNESSetOptionsPrefix(subsnes,prefix);<br> }<br> }<br> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);</font><br></div></div><div><font face="monospace"> ... </font></div><div><font face="monospace"> ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);</font></div><div><br></div><div><span style="font-family:arial,sans-serif">However, I still cannot change SNES, KSP, and PC types for individual domains through the command arguments. I checked the subdomains with </span>-snes_view ::ascii_info_detail and it does show that the prefixes are properly changed. It also shows that the SNES type for the last domain was successfully changed. But f<span style="font-family:arial,sans-serif">or some reason, I only have access to the SNES viewer options during runtime. For example, if I</span><font face="arial, sans-serif"> run</font><font face="monospace"> mpiexec -n 4 ./test1 -sub_0_ksp_type gmres -help | grep sub_0 </font><font face="arial, sans-serif">I get the output: </font></div><div><font face="arial, sans-serif"><br></font></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><font face="monospace">Viewer (-sub_0_snes_convergence_estimate) options:<br> -sub_0_snes_convergence_estimate ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_convergence_estimate binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_convergence_estimate draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_convergence_estimate socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_convergence_estimate saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_view_pre) options:<br> -sub_0_snes_view_pre ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_view_pre binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_view_pre draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_view_pre socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_view_pre saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_test_jacobian_view) options:<br> -sub_0_snes_test_jacobian_view ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_view binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_view draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_view socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_view saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_test_jacobian_display) options:<br> -sub_0_snes_test_jacobian_display ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_display binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_display draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_display socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_test_jacobian_display saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_ksp_converged_reason) options:<br> -sub_0_ksp_converged_reason ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_ksp_converged_reason binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_ksp_converged_reason draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_ksp_converged_reason socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_ksp_converged_reason saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_converged_reason) options:<br> -sub_0_snes_converged_reason ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_converged_reason binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_converged_reason draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_converged_reason socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_converged_reason saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_view) options:<br> -sub_0_snes_view ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_view binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_view draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_view socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_view saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Viewer (-sub_0_snes_view_solution) options:<br> -sub_0_snes_view_solution ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br> -sub_0_snes_view_solution binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br> -sub_0_snes_view_solution draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br> -sub_0_snes_view_solution socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br> -sub_0_snes_view_solution saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>Option left: name:-sub_0_ksp_type value: gmres<br></font></blockquote><div><br></div>Do you know what could be causing this? <br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 17, 2022 at 10:00 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> MPI_Comm_size(<span style="font-family:monospace">PETSC_COMM_WORLD,&size);</span><div><font face="monospace"> MPI_Comm_rank(</font><span style="font-family:monospace">PETSC_COMM_WORLD,&rank);</span></div><div><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><font face="monospace">SNESNASMGetSNES(snes,0,&subsnes);<br> char prefix[10];<br> sprintf(prefix,"sub_%d_",rank);<br> SNESSetOptionsPrefix(subsnes,prefix);</font></div></div></div></blockquote></div></div></blockquote><div><br></div><br><div><br><blockquote type="cite"><div>On Jun 17, 2022, at 9:35 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr">On Fri, Jun 17, 2022 at 9:22 AM Takahashi, Tadanaga <<a href="mailto:tt73@njit.edu" target="_blank">tt73@njit.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I'm having some trouble pulling out the subsolver. I tried to use SNESNASMGetSNES in a loop over each subdomain. However I get an error when I run the code with more than one MPI processors. Here is a snippet from my code: </div><div><br><font face="monospace"> SNES snes, subsnes;<br> PetscMPIInt rank, size;<br></font></div><div><font face="monospace"> ... </font></div><div><div><font face="monospace"> ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);<br> ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);<br> ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);<br> ... <br> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);<br> ierr = SNESSetUp(snes); CHKERRQ(ierr);<br> PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);<br> for (i=0; i<size; i++) {<br> PetscPrintf(PETSC_COMM_WORLD, "rank = %d\n",i);<br> SNESNASMGetSNES(snes,i,&subsnes);<br> // char prefix[10];<br> // sprintf(prefix,"sub_%d_",i);<br> // SNESSetOptionsPrefix(subsnes,prefix);<br> }</font><br></div></div><div><font face="monospace"> ... </font></div><div><font face="monospace"> ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);<br></font></div><div><font face="monospace"><br></font></div><div><br></div><div>And, here is the output of the code when I run with 2 MPI procs: </div></div></blockquote><div><br></div><div><span style="font-family:monospace">SNESNASMGetSNES() gets the local subsolvers. It seems you only have one per process.</span><br></div><div><span style="font-family:monospace">You can check </span><a href="https://petsc.org/main/docs/manualpages/SNES/SNESNASMGetNumber/" target="_blank">https://petsc.org/main/docs/manualpages/SNES/SNESNASMGetNumber/</a></div><div><br></div><div>Notice that your current code will not work because, according to your explanation, you only want to change</div><div>the prefix on a single rank, so you need to check the rank when you do it.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><font face="monospace">takahashi@ubuntu:~/Desktop/MA-DDM/C/Rectangle$ mpiexec -n 2 ./test1<br>Size = 2<br>rank = 0<br>rank = 1<br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Argument out of range<br>[0]PETSC ERROR: No such subsolver<br>[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.17.1, unknown <br>[0]PETSC ERROR: ./test1 on a linux-gnu-c-debug named ubuntu by takahashi Fri Jun 17 06:06:38 2022<br>[0]PETSC ERROR: Configure options --with-mpi-dir=/usr --with-fc=0<br>[0]PETSC ERROR: #1 SNESNASMGetSNES() at /home/takahashi/Desktop/petsc/src/snes/impls/nasm/nasm.c:923<br><br>===================================================================================<br>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>= RANK 0 PID 976566 RUNNING AT ubuntu<br>= KILLED BY SIGNAL: 9 (Killed)<br>===================================================================================</font><br></div><div><br></div><div>This error doesn't occur when I run this without MPI. However, I tried to change the prefix of the subdomain to `sub_0_` but I am not able to change the snes_type using this prefix. Running<font face="monospace"> ./test1 -snes_view -help | grep sub_0_snes_type </font><font face="arial, sans-serif">prints nothing. </font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 16, 2022 at 6:23 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Jun 16, 2022 at 5:57 PM tt73 <<a href="mailto:tt73@njit.edu" target="_blank">tt73@njit.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"><br></div><div dir="auto">Hi, </div><div dir="auto"><br></div><div dir="auto">I am using NASM as the outer solver for a nonlinear problem. For one of the subdomains, I want to run the local solve with a different set of options form the others. Is there any way to set options for each subdomain? </div></div></blockquote></div><br clear="all"><div>I can see two ways:</div><div><br></div><div> 1) Pull out the subsolver and set it using the API</div><div><br></div><div> 2) Pull out the subsolver and give it a different prefix</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>