<div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">Hi,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">Luckily, I fixed my issue. I don't know exactly why I got this error.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">I changed the setting of the Fortran project, do not use the option "Use Intel Math Kernel Library", and it is fine.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">There are possibly some conflicts between the options that were used to build the PETSc library with the Intel MKL library,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">and they cause this problem for vector scattering using PETSc's functions.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">So, I don't need to send you my code.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">Thank you very much for your consideration.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif;font-size:small">Hoang</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 3 Nov 2020 at 12:17, Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div> Hoang,<div><br></div><div> Could you please send the entire code as a .F90 file as an attachment so we can reproduce the problem?</div><div><br><div> Thanks</div><div><br></div><div> Barry</div><div><br><blockquote type="cite"><div>On Nov 2, 2020, at 8:38 PM, Hien HN- <<a href="mailto:hoangnhanhien@gmail.com" target="_blank">hoangnhanhien@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small">Hi Matt,</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small">1) As I understood, the VecScatterCreateToAll function will create the vector sol_to_all, and it does not need to create this vector with, for example, VecCreate or VecCreateMPI function. Is this correct?</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px"> call VecScatterCreateToAll(petsc_</span><span style="font-family:Calibri,sans-serif;font-size:14.6667px">solution,ctx,sol_to_all,petsc_</span><span style="font-family:Calibri,sans-serif;font-size:14.6667px">err)</span> <br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small">I have used the VecView to print the vector sol_to_all to a file. It seems no problem with sol_to_all.</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small">I installed PETSc with Intel MKL and MS-MPI libraries. So should I enable the option of Intel MKL that performs vector operations in my Fortran project?</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small">2) This system is not the <span style="font-family:Calibri,sans-serif;font-size:14.6667px">Laplacian.</span></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px">Thank you very much,</span></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif;font-size:small"><span style="font-family:Calibri,sans-serif;font-size:14.6667px">Hoang</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 2 Nov 2020 at 20:15, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Nov 2, 2020 at 12:31 AM Hien HN- <<a href="mailto:hoangnhanhien@gmail.com" target="_blank">hoangnhanhien@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif;font-size:small"><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Dear PETSc developers,</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>I have used KSP solver to solve a linear equation system in parallel with 5 processors.</b></div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>The linear system consists of a pressure matrix (pressure_coeff_mat) and its RHS (p_coeff_mat) whose values are set every time step using MatSetValue and VecSetValues functions and then assembled using:</b></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call MatAssemblyBegin(pressure_coeff_mat,mat_final_assembly,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call MatAssemblyEnd(pressure_coeff_mat,mat_final_assembly,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecAssemblyBegin(p_coeff_rhs,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecAssemblyEnd(p_coeff_rhs,petsc_err)</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> </p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>One processor solves this linear system using KSBCGS Krylov solver with ILU preconditioner, and the result is scattered to all processors</b></div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call KSPGetPC(petsc_ksp,petsc_pc,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> if (preconditioner==0) then</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call PCSetType(petsc_pc,PCJACOBI,petsc_err</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call PetscOptionsSetValue(PETSC_NULL_OPTIONS,"sub_ksp_type","preonly",petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call PetscOptionsSetValue(PETSC_NULL_OPTIONS,"sub_pc_type","ilu",petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call PCSetFromOptions(petsc_pc,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> else</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call PCSetType(petsc_pc,PCBJACOBI,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> end if</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call KSPSetFromOptions(petsc_ksp,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call KSPSetUp(petsc_ksp,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call KSPSolve(petsc_ksp,p_coeff_rhs,petsc_solution,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecScatterCreateToAll(petsc_solution,ctx,sol_to_all,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecScatterBegin(ctx,petsc_solution,sol_to_all,INSERT_VALUES,SCATTER_FORWARD,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecScatterEnd(ctx,petsc_solution,sol_to_all, INSERT_VALUES,SCATTER_FORWARD,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecGetValues(sol_to_all,ncells, loc_id_to_global-1,sol, petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecScatterDestroy(ctx,petsc_err)</div><div style="margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calibri,sans-serif"> call VecDestroy(sol_to_all,petsc_err)</div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>However, I got the following errors with vector scattering, i.e., VecScatterBegin, VecScatterEnd, and also VecGetValues. I have checked the pressure matrix, its RHS, and the solution for the first call of KSPSolve. Their values seem to be okay (see attached file). So, could you please tell me what does this error mean and how can I fix it.</b></div></div></div></blockquote><div><br></div><div>1) The error message says that '<span style="font-family:Calibri,sans-serif;font-size:14.6667px">sol_to_all' has not been created</span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px">2) Is this system the Laplacian? If so, ILU is a bad preconditioner. You should try gamg.</span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px"> Thanks,</span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="font-family:Calibri,sans-serif;font-size:14.6667px"> Matt</span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif;font-size:small"><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b style="font-size:11pt"> </b></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>Thank you,</b></div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><b>Error message:</b></div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">-----------------------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Configure options --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-blaslapack-lib="[/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_intel_c_dll.lib,/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_sequential_dll.lib,/cygdrive/c/PROGRA~2/Intel/COMPOS~1/mkl/lib/ia32/mkl_core_dll.lib]" --with-mpi-lib="[/cygdrive/c/PROGRA~2/MICROS~1/MPI/Lib/x86/msmpifec.lib,/cygdrive/c/PROGRA~2/MICROS~1/MPI/Lib/x86/msmpi.lib]" --with-mpi-include="[/cygdrive/c/PROGRA~2/MICROS~1/MPI/Include,/cygdrive/c/PROGRA~2/MICROS~1/MPI/Include/x86]" --with-mpiexec=/cygdrive/c/PROGRA~1/MI564C~1/bin/mpiexec.exe --with-debugging=1 --with-shared-libraries=0 --CFLAGS=-Od --FFLAGS=-Od CXXFLAGS=-Od --with-clean=1</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[1]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[2]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[3]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[0]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[4]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#4930 VecScatterBegin() line 83 in C:\petsc\PE4D24~1.13-\src\vec\vscat\INTERF~1\vscatfce.c</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">--------------------- Error Message --------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[1]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[2]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[3]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[0]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[4]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">--------------------- Error Message --------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">--------------------- Error Message --------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">--------------------- Error Message --------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">--------------------- Error Message --------------------------------------------------------------</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null argument, when expecting valid pointer</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[1]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[2]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[3]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[0]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[4]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null argument, when expecting valid pointer</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null argument, when expecting valid pointer</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null argument, when expecting valid pointer</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null argument, when expecting valid pointer</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null Object: Parameter # 3</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[1]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[2]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[3]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[0]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[4]PETSC ERROR:</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null Object: Parameter # 3</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null Object: Parameter # 3</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null Object: Parameter # 3</div><div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Null Object: Parameter # 3</div></div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Hoang Nhan Hien, Ph.D.<div><br></div><div>1) Nuclear Safety Department, Institute for Nuclear Science and Technology, Vietnam Atomic Energy Insitute</div><div><br></div><div>2) Nuclear Safety Analysis Method Development, Enviroment & Energy Technology, Inc.</div><div><br></div><div>Email: <a href="mailto:hoangnhanhien@gmail.com" target="_blank">hoangnhanhien@gmail.com</a>; <a href="mailto:hnhien@kaeri.re.kr" target="_blank">h</a><a href="mailto:ien@en2t.com" target="_blank">ien@en2t.com</a><br></div><div>Phone: +81-10-2635-2411</div><div><br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Hoang Nhan Hien, Ph.D.<div><br></div><div>1) Nuclear Safety Department, Institute for Nuclear Science and Technology, Vietnam Atomic Energy Insitute</div><div><br></div><div>2) Nuclear Safety Analysis Method Development, Enviroment & Energy Technology, Inc.</div><div><br></div><div>Email: <a href="mailto:hoangnhanhien@gmail.com" target="_blank">hoangnhanhien@gmail.com</a>; <a href="mailto:hnhien@kaeri.re.kr" target="_blank">h</a><a href="mailto:ien@en2t.com" target="_blank">ien@en2t.com</a><br></div><div>Phone: +81-10-2635-2411</div><div><br></div></div></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Hoang Nhan Hien, Ph.D.<div><br></div><div>1) Nuclear Safety Department, Institute for Nuclear Science and Technology, Vietnam Atomic Energy Insitute</div><div><br></div><div>2) Nuclear Safety Analysis Method Development, Enviroment & Energy Technology, Inc.</div><div><br></div><div>Email: <a href="mailto:hoangnhanhien@gmail.com" target="_blank">hoangnhanhien@gmail.com</a>; <a href="mailto:hnhien@kaeri.re.kr" target="_blank">h</a><a href="mailto:ien@en2t.com" target="_blank">ien@en2t.com</a><br></div><div>Phone: +81-10-2635-2411</div><div><br></div></div></div></div></div>