<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Hoang,<div class=""><br class=""></div><div class=""> Could you please send the entire code as a .F90 file as an attachment so we can reproduce the problem?</div><div class=""><br class=""><div> Thanks</div><div><br class=""></div><div> Barry</div><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 2, 2020, at 8:38 PM, Hien HN- <<a href="mailto:hoangnhanhien@gmail.com" class="">hoangnhanhien@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">Hi Matt,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">1) As I understood, the VecScatterCreateToAll function will create the vector sol_to_all, and it does not need to create this vector with, for example, VecCreate or VecCreateMPI function. Is this correct?</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""> call VecScatterCreateToAll(petsc_</span><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">solution,ctx,sol_to_all,petsc_</span><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">err)</span> <br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">I have used the VecView to print the vector sol_to_all to a file. It seems no problem with sol_to_all.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">I installed PETSc with Intel MKL and MS-MPI libraries. So should I enable the option of Intel MKL that performs vector operations in my Fortran project?</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br class=""></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">2) This system is not the <span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">Laplacian.</span></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""><br class=""></span></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">Thank you very much,</span></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">Hoang</span></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 2 Nov 2020 at 20:15, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Mon, Nov 2, 2020 at 12:31 AM Hien HN- <<a href="mailto:hoangnhanhien@gmail.com" target="_blank" class="">hoangnhanhien@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family:"trebuchet ms",sans-serif;font-size:small" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Dear PETSc developers,</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">I have used KSP solver to solve a linear equation system in parallel with 5 processors.</b></div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">The linear system consists of a pressure matrix (pressure_coeff_mat) and its RHS (p_coeff_mat) whose values are set every time step using MatSetValue and VecSetValues functions and then assembled using:</b></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatAssemblyBegin(pressure_coeff_mat,mat_final_assembly,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatAssemblyEnd(pressure_coeff_mat,mat_final_assembly,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecAssemblyBegin(p_coeff_rhs,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecAssemblyEnd(p_coeff_rhs,petsc_err)</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> </p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">One processor solves this linear system using KSBCGS Krylov solver with ILU preconditioner, and the result is scattered to all processors</b></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call KSPGetPC(petsc_ksp,petsc_pc,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> if (preconditioner==0) then</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call PCSetType(petsc_pc,PCJACOBI,petsc_err</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call PetscOptionsSetValue(PETSC_NULL_OPTIONS,"sub_ksp_type","preonly",petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call PetscOptionsSetValue(PETSC_NULL_OPTIONS,"sub_pc_type","ilu",petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call PCSetFromOptions(petsc_pc,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> else</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call PCSetType(petsc_pc,PCBJACOBI,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> end if</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call KSPSetFromOptions(petsc_ksp,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call KSPSetUp(petsc_ksp,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call KSPSolve(petsc_ksp,p_coeff_rhs,petsc_solution,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecScatterCreateToAll(petsc_solution,ctx,sol_to_all,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecScatterBegin(ctx,petsc_solution,sol_to_all,INSERT_VALUES,SCATTER_FORWARD,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecScatterEnd(ctx,petsc_solution,sol_to_all, INSERT_VALUES,SCATTER_FORWARD,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecGetValues(sol_to_all,ncells, loc_id_to_global-1,sol, petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecScatterDestroy(ctx,petsc_err)</div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call VecDestroy(sol_to_all,petsc_err)</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">However, I got the following errors with vector scattering, i.e., VecScatterBegin, VecScatterEnd, and also VecGetValues. I have checked the pressure matrix, its RHS, and the solution for the first call of KSPSolve. Their values seem to be okay (see attached file). So, could you please tell me what does this error mean and how can I fix it.</b></div></div></div></blockquote><div class=""><br class=""></div><div class="">1) The error message says that '<span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">sol_to_all' has not been created</span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""><br class=""></span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class="">2) Is this system the Laplacian? If so, ILU is a bad preconditioner. You should try gamg.</span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""><br class=""></span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""> Thanks,</span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""><br class=""></span></div><div class=""><span style="font-family:Calibri,sans-serif;font-size:14.6667px" class=""> Matt</span></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div style="font-family:"trebuchet ms",sans-serif;font-size:small" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b style="font-size:11pt" class=""> </b></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Thank you,</b></div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Error message:</b></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">-----------------------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Configure options --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-blaslapack-lib="[/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_intel_c_dll.lib,/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_sequential_dll.lib,/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_core_dll.lib]" --with-mpi-lib="[/cygdrive/c/PROGRA~2/MICROS~1/MPI/Lib/x86/msmpifec.lib,/cygdrive/c/PROGRA~2/MICROS~1/MPI/Lib/x86/msmpi.lib]" --with-mpi-include="[/cygdrive/c/PROGRA~2/MICROS~1/MPI/Include,/cygdrive/c/PROGRA~2/MICROS~1/MPI/Include/x86]" --with-mpiexec=/cygdrive/c/PROGRA~1/MI564C~1/bin/mpiexec.exe --with-debugging=1 --with-shared-libraries=0 --CFLAGS=-Od --FFLAGS=-Od CXXFLAGS=-Od --with-clean=1</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[2]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[3]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[4]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------- Error Message --------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[2]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[3]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[4]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------- Error Message --------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------- Error Message --------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------- Error Message --------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------- Error Message --------------------------------------------------------------</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null argument, when expecting valid pointer</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[2]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[3]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[4]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null argument, when expecting valid pointer</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null argument, when expecting valid pointer</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null argument, when expecting valid pointer</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null argument, when expecting valid pointer</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null Object: Parameter # 3</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[2]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[3]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[4]PETSC ERROR:</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null Object: Parameter # 3</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null Object: Parameter # 3</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null Object: Parameter # 3</div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Null Object: Parameter # 3</div></div><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class="">Hoang Nhan Hien, Ph.D.<div class=""><br class=""></div><div class="">1) Nuclear Safety Department, Institute for Nuclear Science and Technology, Vietnam Atomic Energy Insitute</div><div class=""><br class=""></div><div class="">2) Nuclear Safety Analysis Method Development, Enviroment & Energy Technology, Inc.</div><div class=""><br class=""></div><div class="">Email: <a href="mailto:hoangnhanhien@gmail.com" target="_blank" class="">hoangnhanhien@gmail.com</a>; <a href="mailto:hnhien@kaeri.re.kr" target="_blank" class="">h</a><a href="mailto:ien@en2t.com" target="_blank" class="">ien@en2t.com</a><br class=""></div><div class="">Phone: +81-10-2635-2411</div><div class=""><br class=""></div></div></div></div></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class="">Hoang Nhan Hien, Ph.D.<div class=""><br class=""></div><div class="">1) Nuclear Safety Department, Institute for Nuclear Science and Technology, Vietnam Atomic Energy Insitute</div><div class=""><br class=""></div><div class="">2) Nuclear Safety Analysis Method Development, Enviroment & Energy Technology, Inc.</div><div class=""><br class=""></div><div class="">Email: <a href="mailto:hoangnhanhien@gmail.com" target="_blank" class="">hoangnhanhien@gmail.com</a>; <a href="mailto:hnhien@kaeri.re.kr" target="_blank" class="">h</a><a href="mailto:ien@en2t.com" target="_blank" class="">ien@en2t.com</a><br class=""></div><div class="">Phone: +81-10-2635-2411</div><div class=""><br class=""></div></div></div></div></div>
</div></blockquote></div><br class=""></div></body></html>