From bsmith at petsc.dev Sat Jul 1 11:16:36 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 1 Jul 2023 12:16:36 -0400 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> Message-ID: <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> I see no reason not to trust the times below, they seem reasonable. You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 to 256. The total amount of data moved (number of messages moved times average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 to 2.3000e+09. A pretty moderate amount of data increase, but note that each time you double the number of ranks, you also increase substantially the network's hardware to move data, so one would hope for a good speed up. Also, the load balance is very good, near 1. Often with assembly, we see very out-of-balance, and it is difficult to get good speedup when the balance is really off. It looks like over 90% of the entire run time is coming from setting and assembling the values? Also the setting values time dominates assembly time more with more ranks. Are you setting a single value at a time or a collection of them? How big are the vectors? Run all three cases with -info :vec to see some information about how many mallocs where move to hold the stashed vector entries. > On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: > > > > Hi, > Thanks for your reply. I try to use PetscLogEvent(), and the result shows same conclusion. > What I have done is : > ---------------- > PetscLogEvent Mat_assemble_event, Mat_setvalue_event, Mat_setAsse_event; > PetscClassId classid; > PetscLogDouble user_event_flops; > PetscClassIdRegister("Test assemble and set value", &classid); > PetscLogEventRegister("Test only assemble", classid, &Mat_assemble_event); > PetscLogEventRegister("Test only set values", classid, &Mat_setvalue_event); > PetscLogEventRegister("Test both assemble and set values", classid, &Mat_setAsse_event); > PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); > PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); > ...compute elements and use MatSetValues. No call for assembly > PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); > > PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); > PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); > ---------------- > > And the output as follows. By the way, dose petsc recorde all time between PetscLogEventBegin and PetscLogEventEnd? or just test the time of petsc API? It is all of the time. > ---------------- > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > 64new 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 > 128new 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 > 256new 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 > > 64: > only assemble 1 1.0 2.6596e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 > only setvalues 1 1.0 1.9987e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > Test both 1 1.0 4.6580e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 > > 128: > only assemble 1 1.0 6.9718e+01 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 > only setvalues 1 1.0 1.4438e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 > Test both 1 1.0 2.1417e+02 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 > > 256: > only assemble 1 1.0 1.7482e+01 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 > only setvalues 1 1.0 1.3717e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 > Test both 1 1.0 1.5475e+02 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 > > > > Runfeng > > Barry Smith > ?2023?6?30??? 23:35??? >> >> You cannot look just at the VecAssemblyEnd() time, that will very likely give the wrong impression of the total time it takes to put the values in. >> >> You need to register a new Event and put a PetscLogEvent() just before you start generating the vector entries and calling VecSetValues() and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way to get an accurate accounting of the time. >> >> Barry >> >> >> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin > wrote: >> > >> > Hello! >> > >> > When I use PETSc build a sbaij matrix, I find a strange thing. When I increase the number of processors, the assemble time become smaller. All these are totally same matrix. The assemble time mainly arouse from message passing, which because I use dynamic workload that it is random for which elements are computed by which processor. >> > But from instinct, if use more processors, then more possible that the processor computes elements storing in other processors. But from the output of log_view, It seems when use more processors, the processors compute more elements storing in its local(infer from that, with more processors, less total amount of passed messages). >> > >> > What could cause this happened? Thank you! >> > >> > >> > Following is the output of log_view for 64\128\256 processors. Every row is time profiler of VecAssemblyEnd. >> > >> > ------------------------------------------------------------------------------------------------------------------------ >> > processors Count Time (sec) Flop --- Global --- --- Stage ---- Total >> > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> > 64 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >> > 128 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >> > 256 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >> > >> > Runfeng Jin >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jul 1 11:16:36 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 1 Jul 2023 12:16:36 -0400 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> Message-ID: <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> I see no reason not to trust the times below, they seem reasonable. You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 to 256. The total amount of data moved (number of messages moved times average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 to 2.3000e+09. A pretty moderate amount of data increase, but note that each time you double the number of ranks, you also increase substantially the network's hardware to move data, so one would hope for a good speed up. Also, the load balance is very good, near 1. Often with assembly, we see very out-of-balance, and it is difficult to get good speedup when the balance is really off. It looks like over 90% of the entire run time is coming from setting and assembling the values? Also the setting values time dominates assembly time more with more ranks. Are you setting a single value at a time or a collection of them? How big are the vectors? Run all three cases with -info :vec to see some information about how many mallocs where move to hold the stashed vector entries. > On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: > > > > Hi, > Thanks for your reply. I try to use PetscLogEvent(), and the result shows same conclusion. > What I have done is : > ---------------- > PetscLogEvent Mat_assemble_event, Mat_setvalue_event, Mat_setAsse_event; > PetscClassId classid; > PetscLogDouble user_event_flops; > PetscClassIdRegister("Test assemble and set value", &classid); > PetscLogEventRegister("Test only assemble", classid, &Mat_assemble_event); > PetscLogEventRegister("Test only set values", classid, &Mat_setvalue_event); > PetscLogEventRegister("Test both assemble and set values", classid, &Mat_setAsse_event); > PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); > PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); > ...compute elements and use MatSetValues. No call for assembly > PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); > > PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); > PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); > ---------------- > > And the output as follows. By the way, dose petsc recorde all time between PetscLogEventBegin and PetscLogEventEnd? or just test the time of petsc API? It is all of the time. > ---------------- > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > 64new 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 > 128new 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 > 256new 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 > > 64: > only assemble 1 1.0 2.6596e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 > only setvalues 1 1.0 1.9987e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > Test both 1 1.0 4.6580e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 > > 128: > only assemble 1 1.0 6.9718e+01 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 > only setvalues 1 1.0 1.4438e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 > Test both 1 1.0 2.1417e+02 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 > > 256: > only assemble 1 1.0 1.7482e+01 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 > only setvalues 1 1.0 1.3717e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 > Test both 1 1.0 1.5475e+02 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 > > > > Runfeng > > Barry Smith > ?2023?6?30??? 23:35??? >> >> You cannot look just at the VecAssemblyEnd() time, that will very likely give the wrong impression of the total time it takes to put the values in. >> >> You need to register a new Event and put a PetscLogEvent() just before you start generating the vector entries and calling VecSetValues() and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way to get an accurate accounting of the time. >> >> Barry >> >> >> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin > wrote: >> > >> > Hello! >> > >> > When I use PETSc build a sbaij matrix, I find a strange thing. When I increase the number of processors, the assemble time become smaller. All these are totally same matrix. The assemble time mainly arouse from message passing, which because I use dynamic workload that it is random for which elements are computed by which processor. >> > But from instinct, if use more processors, then more possible that the processor computes elements storing in other processors. But from the output of log_view, It seems when use more processors, the processors compute more elements storing in its local(infer from that, with more processors, less total amount of passed messages). >> > >> > What could cause this happened? Thank you! >> > >> > >> > Following is the output of log_view for 64\128\256 processors. Every row is time profiler of VecAssemblyEnd. >> > >> > ------------------------------------------------------------------------------------------------------------------------ >> > processors Count Time (sec) Flop --- Global --- --- Stage ---- Total >> > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> > 64 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >> > 128 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >> > 256 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >> > >> > Runfeng Jin >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wanghemeng at 163.com Sun Jul 2 00:47:14 2023 From: wanghemeng at 163.com (=?UTF-8?B?546L6LWr6JCM?=) Date: Sun, 2 Jul 2023 13:47:14 +0800 (GMT+08:00) Subject: [petsc-users] Question about residue norm in PETSc Message-ID: Dear PETSc Team, Sorry to bother! My name is Hemeng Wang, and I am currently learning the use of PETSc software package. I am confused while calculating the norm of residue. I calculated residue norm by myself with: ``` PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) PetscCall(MatMult(A, x, u)); PetscCall(VecAXPY(b, -1.0, u)); PetscCall(VecNorm(b, NORM_2, &norm_delta)); ``` and check the (norm_delta) / (norm_b). It seems not the `atol` which set by `KSPSetTolerances()`. (I set atol as 1e-12, but got 5e-7 finally) (options: -ksp_type cg -pc_type gamg -ksp_converged_reason -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) I also check the soure code of `KSPSolve_CG` in `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the difference. I will really really appreciated if someone can explain the calculation of `resid norm` in petsc. And where is my mistake. To provide you with more context, here are the source code about my implementation. And the output of my test. main.c Main code of my program mmio.h mmloader.h Headers for matrix read Makefile For compiling, same as sample provided task.sh A script for running program in `slurm` slurm-4803840.out Output of my test Thank you very much for your time and attention. I greatly appreciate your support and look forward to hearing from you soon. Best regards, Hemeng Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: main.c URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 818 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: mmio.h URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: mmloader.h URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: slurm-4803840.out Type: application/octet-stream Size: 13311 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: task.sh Type: application/x-sh Size: 585 bytes Desc: not available URL: From knepley at gmail.com Sun Jul 2 05:50:43 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Jul 2023 06:50:43 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: References: Message-ID: On Sun, Jul 2, 2023 at 2:24?AM ??? wrote: > Dear PETSc Team, > > Sorry to bother! My name is Hemeng Wang, and I am currently learning the > use of PETSc software package. I am confused while calculating the norm of > residue. > > I calculated residue norm by myself with: > ``` > PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) > > PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) > PetscCall(MatMult(A, x, u)); > PetscCall(VecAXPY(b, -1.0, u)); > PetscCall(VecNorm(b, NORM_2, &norm_delta)); > ``` > and check the (norm_delta) / (norm_b). It seems not the `atol` which set > by `KSPSetTolerances()`. > (I set atol as 1e-12, but got 5e-7 finally) > (options: -ksp_type cg -pc_type gamg -ksp_converged_reason -ksp_norm_type > unpreconditioned -ksp_monitor_true_residual) > If you are using the default convergence test, there is an absolute tolerance (atol) _and_ a relative tolerance (rtol). It seems likely you hit the relative tolerance. You can check this using -ksp_converged_reason You could make rtol really small if you want to just see the atol -ksp_rtol 1e-20 Thanks, Matt > I also check the soure code of `KSPSolve_CG` in > `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the > difference. > > I will really really appreciated if someone can explain the calculation of > `resid norm` in petsc. And where is my mistake. > > To provide you with more context, here are the source code about my > implementation. And the output of my test. > > main.c > Main code of my program > > mmio.h mmloader.h > Headers for matrix read > > Makefile > For compiling, same as sample provided > > task.sh > A script for running program in `slurm` > > slurm-4803840.out > Output of my test > > Thank you very much for your time and attention. I greatly appreciate your > support and look forward to hearing from you soon. > Best regards, > Hemeng Wang > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 2 07:05:19 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Jul 2023 08:05:19 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: <6e8757e9.3411.1891672f933.Coremail.wanghemeng@163.com> References: <6e8757e9.3411.1891672f933.Coremail.wanghemeng@163.com> Message-ID: On Sun, Jul 2, 2023 at 7:53?AM ??? wrote: > Thanks for your reply! > So sorry that I made a mistake in the description. > I set the tolerances by: > PetscCall(KSPSetTolerances(ksp, 1e-12, DBL_MIN, PETSC_DEFAULT, > PETSC_DEFAULT)); > and got (by passing `-ksp_norm_type unpreconditioned > -ksp_monitor_true_residual`) > 74 KSP unpreconditioned resid norm 7.256655641876e-08 true resid norm > 7.256655641876e-08 ||r(i)||/||b|| 9.975738158726e-13 > I'm wondering why the ` ||r(i)||/||b||` is different with mine which > calculated by: > ``` > PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) (which is > 72743.044439) > PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) > PetscCall(MatMult(A, x, u)); > PetscCall(VecAXPY(b, -1.0, u)); > PetscCall(VecNorm(b, NORM_2, &norm_delta)); // (which is 0.039608) > ``` > and (norm_delta) / (norm_b) = 5.44496e-07 which is higher and different > with the rtol I set (1e-12). > Sorry again for the waste of your time. I would really appreciated if you > could help me again! > 1) 7.256655641876e-08 / 72743.044439 = 9.975738158726e-13 so ||r_i||/||b|| is correct in the output 2) You are asking why you calculate a different residual? I will have to run your code. Thanks, Matt > Best regards! > Hemeng Wang > > > ---- Replied Message ---- > From Matthew Knepley > Date 7/2/2023 18:51 > To ??? > Cc petsc-users at mcs.anl.gov > > Subject Re: [petsc-users] Question about residue norm in PETSc > On Sun, Jul 2, 2023 at 2:24?AM ??? wrote: > >> Dear PETSc Team, >> >> Sorry to bother! My name is Hemeng Wang, and I am currently learning the >> use of PETSc software package. I am confused while calculating the norm of >> residue. >> >> I calculated residue norm by myself with: >> ``` >> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) >> >> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >> PetscCall(MatMult(A, x, u)); >> PetscCall(VecAXPY(b, -1.0, u)); >> PetscCall(VecNorm(b, NORM_2, &norm_delta)); >> ``` >> and check the (norm_delta) / (norm_b). It seems not the `atol` which set >> by `KSPSetTolerances()`. >> (I set atol as 1e-12, but got 5e-7 finally) >> (options: -ksp_type cg -pc_type gamg -ksp_converged_reason >> -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) >> > > If you are using the default convergence test, there is an absolute > tolerance (atol) _and_ a relative tolerance (rtol). It seems likely you hit > the relative tolerance. You can check this using > > -ksp_converged_reason > > You could make rtol really small if you want to just see the atol > > -ksp_rtol 1e-20 > > Thanks, > > Matt > > >> I also check the soure code of `KSPSolve_CG` in >> `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the >> difference. >> >> I will really really appreciated if someone can explain the calculation >> of `resid norm` in petsc. And where is my mistake. >> >> To provide you with more context, here are the source code about my >> implementation. And the output of my test. >> >> main.c >> Main code of my program >> >> mmio.h mmloader.h >> Headers for matrix read >> >> Makefile >> For compiling, same as sample provided >> >> task.sh >> A script for running program in `slurm` >> >> slurm-4803840.out >> Output of my test >> >> Thank you very much for your time and attention. I greatly appreciate >> your support and look forward to hearing from you soon. >> Best regards, >> Hemeng Wang >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 2 07:10:31 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Jul 2023 08:10:31 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: References: <6e8757e9.3411.1891672f933.Coremail.wanghemeng@163.com> Message-ID: On Sun, Jul 2, 2023 at 8:05?AM Matthew Knepley wrote: > On Sun, Jul 2, 2023 at 7:53?AM ??? wrote: > >> Thanks for your reply! >> So sorry that I made a mistake in the description. >> I set the tolerances by: >> PetscCall(KSPSetTolerances(ksp, 1e-12, DBL_MIN, PETSC_DEFAULT, >> PETSC_DEFAULT)); >> and got (by passing `-ksp_norm_type unpreconditioned >> -ksp_monitor_true_residual`) >> 74 KSP unpreconditioned resid norm 7.256655641876e-08 true resid norm >> 7.256655641876e-08 ||r(i)||/||b|| 9.975738158726e-13 >> I'm wondering why the ` ||r(i)||/||b||` is different with mine which >> calculated by: >> ``` >> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) (which is >> 72743.044439) >> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >> PetscCall(MatMult(A, x, u)); >> PetscCall(VecAXPY(b, -1.0, u)); >> PetscCall(VecNorm(b, NORM_2, &norm_delta)); // (which is 0.039608) >> ``` >> and (norm_delta) / (norm_b) = 5.44496e-07 which is higher and different >> with the rtol I set (1e-12). >> Sorry again for the waste of your time. I would really appreciated if you >> could help me again! >> > > 1) 7.256655641876e-08 / 72743.044439 = 9.975738158726e-13 so ||r_i||/||b|| > is correct in the output > > 2) You are asking why you calculate a different residual? I will have to > run your code. > I built your code, but you did not send the matrix and rhs. I suggest using PetscCall(VecAXPY(u, -1.0, b)); instead so that you do not change b, and keep the residual in u. Thanks, Matt Thanks, Matt > Thanks, > > Matt > > >> Best regards! >> Hemeng Wang >> >> >> ---- Replied Message ---- >> From Matthew Knepley >> Date 7/2/2023 18:51 >> To ??? >> Cc petsc-users at mcs.anl.gov >> >> Subject Re: [petsc-users] Question about residue norm in PETSc >> On Sun, Jul 2, 2023 at 2:24?AM ??? wrote: >> >>> Dear PETSc Team, >>> >>> Sorry to bother! My name is Hemeng Wang, and I am currently learning the >>> use of PETSc software package. I am confused while calculating the norm of >>> residue. >>> >>> I calculated residue norm by myself with: >>> ``` >>> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) >>> >>> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >>> PetscCall(MatMult(A, x, u)); >>> PetscCall(VecAXPY(b, -1.0, u)); >>> PetscCall(VecNorm(b, NORM_2, &norm_delta)); >>> ``` >>> and check the (norm_delta) / (norm_b). It seems not the `atol` which set >>> by `KSPSetTolerances()`. >>> (I set atol as 1e-12, but got 5e-7 finally) >>> (options: -ksp_type cg -pc_type gamg -ksp_converged_reason >>> -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) >>> >> >> If you are using the default convergence test, there is an absolute >> tolerance (atol) _and_ a relative tolerance (rtol). It seems likely you hit >> the relative tolerance. You can check this using >> >> -ksp_converged_reason >> >> You could make rtol really small if you want to just see the atol >> >> -ksp_rtol 1e-20 >> >> Thanks, >> >> Matt >> >> >>> I also check the soure code of `KSPSolve_CG` in >>> `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the >>> difference. >>> >>> I will really really appreciated if someone can explain the calculation >>> of `resid norm` in petsc. And where is my mistake. >>> >>> To provide you with more context, here are the source code about my >>> implementation. And the output of my test. >>> >>> main.c >>> Main code of my program >>> >>> mmio.h mmloader.h >>> Headers for matrix read >>> >>> Makefile >>> For compiling, same as sample provided >>> >>> task.sh >>> A script for running program in `slurm` >>> >>> slurm-4803840.out >>> Output of my test >>> >>> Thank you very much for your time and attention. I greatly appreciate >>> your support and look forward to hearing from you soon. >>> Best regards, >>> Hemeng Wang >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 2 07:32:24 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Jul 2023 08:32:24 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: <7360308b.349a.189168b1831.Coremail.wanghemeng@163.com> References: <6e8757e9.3411.1891672f933.Coremail.wanghemeng@163.com> <7360308b.349a.189168b1831.Coremail.wanghemeng@163.com> Message-ID: On Sun, Jul 2, 2023 at 8:19?AM ??? wrote: > Here is the mat and rhs used in code! (May need to change the data path) > > mat: > > https://studentcupeducn-my.sharepoint.com/:u:/g/personal/wanghemeng_student_cup_edu_cn/Ed76oGtC1ttDriZsObbPR74BCnDPUP8aicVXQEL4sO1AyQ?e=zeszik > rhs: > > https://studentcupeducn-my.sharepoint.com/:u:/g/personal/wanghemeng_student_cup_edu_cn/EdHRqWbzVmtIkAppOLL1UMIBM7tK7ws0gEASESGHuGC3yw?e=SMQSmY > > I tried using > PetscCall(VecAXPY(u, -1.0, b)); > but is just as same as > PetscCall(VecAXPY(b, -1.0, u)); > > Thank you so much for that!!! > 90 KSP unpreconditioned resid norm 9.749157899195e+05 true resid norm 9.749157899195e+05 ||r(i)||/||b|| 1.340218569960e+01 91 KSP unpreconditioned resid norm 1.073123446417e+06 true resid norm 1.073123446417e+06 ||r(i)||/||b|| 1.475224820050e+01 92 KSP unpreconditioned resid norm 1.170251286554e+06 true resid norm 1.170251286554e+06 ||r(i)||/||b|| 1.608746644557e+01 93 KSP unpreconditioned resid norm 1.264719067990e+06 true resid norm 1.264719067990e+06 ||r(i)||/||b|| 1.738611681365e+01 94 KSP unpreconditioned resid norm 1.329446257320e+06 true resid norm 1.329446257320e+06 ||r(i)||/||b|| 1.827592270272e+01 95 KSP unpreconditioned resid norm 1.365944956372e+06 true resid norm 1.365944956372e+06 ||r(i)||/||b|| 1.877767100504e+01 96 KSP unpreconditioned resid norm 1.369513563400e+06 true resid norm 1.369513563400e+06 ||r(i)||/||b|| 1.882672871297e+01 97 KSP unpreconditioned resid norm 1.364905651654e+06 true resid norm 1.364905651654e+06 ||r(i)||/||b|| 1.876338366353e+01 98 KSP unpreconditioned resid norm 1.352584030803e+06 true resid norm 1.352584030803e+06 ||r(i)||/||b|| 1.859399810996e+01 99 KSP unpreconditioned resid norm 1.330589478009e+06 true resid norm 1.330589478009e+06 ||r(i)||/||b|| 1.829163857903e+01 100 KSP unpreconditioned resid norm 1.312782529439e+06 true resid norm 1.312782529439e+06 ||r(i)||/||b|| 1.804684612214e+01 Linear solve did not converge due to DIVERGED_ITS iterations 100 KSPSolve Time: 7579.364000 ms norm_delta 1312782.529439, norm_b 72743.044439 Norm of error 18.0468 iterations 100 Residue of error 1.31278e+06 iterations 100 I ran with -pc_type jacobi -ksp_max_it 100 because GAMG takes a long time to setup on my laptop. Those numbers match exactly. THanks, Matt I'm such a beginner T_T > ---- Replied Message ---- > From Matthew Knepley > Date 7/2/2023 20:10 > To ??? > Cc PETSc > Subject Re: [petsc-users] Question about residue norm in PETSc > On Sun, Jul 2, 2023 at 8:05?AM Matthew Knepley wrote: > >> On Sun, Jul 2, 2023 at 7:53?AM ??? wrote: >> >>> Thanks for your reply! >>> So sorry that I made a mistake in the description. >>> I set the tolerances by: >>> PetscCall(KSPSetTolerances(ksp, 1e-12, DBL_MIN, PETSC_DEFAULT, >>> PETSC_DEFAULT)); >>> and got (by passing `-ksp_norm_type unpreconditioned >>> -ksp_monitor_true_residual`) >>> 74 KSP unpreconditioned resid norm 7.256655641876e-08 true resid norm >>> 7.256655641876e-08 ||r(i)||/||b|| 9.975738158726e-13 >>> I'm wondering why the ` ||r(i)||/||b||` is different with mine which >>> calculated by: >>> ``` >>> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) (which is >>> 72743.044439) >>> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >>> PetscCall(MatMult(A, x, u)); >>> PetscCall(VecAXPY(b, -1.0, u)); >>> PetscCall(VecNorm(b, NORM_2, &norm_delta)); // (which is 0.039608) >>> ``` >>> and (norm_delta) / (norm_b) = 5.44496e-07 which is higher and different >>> with the rtol I set (1e-12). >>> Sorry again for the waste of your time. I would really appreciated if >>> you could help me again! >>> >> >> 1) 7.256655641876e-08 / 72743.044439 = 9.975738158726e-13 so >> ||r_i||/||b|| is correct in the output >> >> 2) You are asking why you calculate a different residual? I will have to >> run your code. >> > > I built your code, but you did not send the matrix and rhs. > > I suggest using > > PetscCall(VecAXPY(u, -1.0, b)); > > instead so that you do not change b, and keep the residual in u. > > Thanks, > > Matt > > Thanks, > > Matt > > >> Thanks, >> >> Matt >> >> >>> Best regards! >>> Hemeng Wang >>> >>> >>> ---- Replied Message ---- >>> From Matthew Knepley >>> Date 7/2/2023 18:51 >>> To ??? >>> Cc petsc-users at mcs.anl.gov >>> >>> Subject Re: [petsc-users] Question about residue norm in PETSc >>> On Sun, Jul 2, 2023 at 2:24?AM ??? wrote: >>> >>>> Dear PETSc Team, >>>> >>>> Sorry to bother! My name is Hemeng Wang, and I am currently learning >>>> the use of PETSc software package. I am confused while calculating the norm >>>> of residue. >>>> >>>> I calculated residue norm by myself with: >>>> ``` >>>> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) >>>> >>>> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >>>> PetscCall(MatMult(A, x, u)); >>>> PetscCall(VecAXPY(b, -1.0, u)); >>>> PetscCall(VecNorm(b, NORM_2, &norm_delta)); >>>> ``` >>>> and check the (norm_delta) / (norm_b). It seems not the `atol` which >>>> set by `KSPSetTolerances()`. >>>> (I set atol as 1e-12, but got 5e-7 finally) >>>> (options: -ksp_type cg -pc_type gamg -ksp_converged_reason >>>> -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) >>>> >>> >>> If you are using the default convergence test, there is an absolute >>> tolerance (atol) _and_ a relative tolerance (rtol). It seems likely you hit >>> the relative tolerance. You can check this using >>> >>> -ksp_converged_reason >>> >>> You could make rtol really small if you want to just see the atol >>> >>> -ksp_rtol 1e-20 >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> I also check the soure code of `KSPSolve_CG` in >>>> `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the >>>> difference. >>>> >>>> I will really really appreciated if someone can explain the calculation >>>> of `resid norm` in petsc. And where is my mistake. >>>> >>>> To provide you with more context, here are the source code about my >>>> implementation. And the output of my test. >>>> >>>> main.c >>>> Main code of my program >>>> >>>> mmio.h mmloader.h >>>> Headers for matrix read >>>> >>>> Makefile >>>> For compiling, same as sample provided >>>> >>>> task.sh >>>> A script for running program in `slurm` >>>> >>>> slurm-4803840.out >>>> Output of my test >>>> >>>> Thank you very much for your time and attention. I greatly appreciate >>>> your support and look forward to hearing from you soon. >>>> Best regards, >>>> Hemeng Wang >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 2 08:20:41 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Jul 2023 09:20:41 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: <10087001.351f.18916a2fbf3.Coremail.wanghemeng@163.com> References: <6e8757e9.3411.1891672f933.Coremail.wanghemeng@163.com> <7360308b.349a.189168b1831.Coremail.wanghemeng@163.com> <10087001.351f.18916a2fbf3.Coremail.wanghemeng@163.com> Message-ID: On Sun, Jul 2, 2023 at 8:45?AM ??? wrote: > Thank so much for your patience! I'm really grateful for that! > > Could you explain the calculation of "1.31278e+06" > It appears that my calculation of this value is "18.0468" > 100 KSP unpreconditioned resid norm 1.312782529439e+06 true resid norm 1.312782529439e+06 ||r(i)||/||b|| 1.804684612214e+01 norm_delta 1312782.529439, norm_b 72743.044439 Norm of error 18.0468 iterations 100 Residue of error 1.31278e+06 iterations 100 Here norm_delta matches true resid norm, because they are both A x - b. What you are calling "Norm of error" is actually the "Relative Residual" ||r_100|| / ||b|| = 18.0468, which is correct since that is 1312782.529439 / 72743.044439. What you call "Residue of error" is the "Residual" just as reported in the KSP monitor output. Thanks, Matt > Please forgive my ignorance. I just begin my learning on iterative method. > > Thank you so much for the reply! > > Welcome to China if you have time! ^_^ > > > > ---- Replied Message ---- > From Matthew Knepley > Date 7/2/2023 20:32 > To ??? > Cc PETSc > Subject Re: [petsc-users] Question about residue norm in PETSc > On Sun, Jul 2, 2023 at 8:19?AM ??? wrote: > >> Here is the mat and rhs used in code! (May need to change the data path) >> >> mat: >> >> https://studentcupeducn-my.sharepoint.com/:u:/g/personal/wanghemeng_student_cup_edu_cn/Ed76oGtC1ttDriZsObbPR74BCnDPUP8aicVXQEL4sO1AyQ?e=zeszik >> rhs: >> >> https://studentcupeducn-my.sharepoint.com/:u:/g/personal/wanghemeng_student_cup_edu_cn/EdHRqWbzVmtIkAppOLL1UMIBM7tK7ws0gEASESGHuGC3yw?e=SMQSmY >> >> I tried using >> PetscCall(VecAXPY(u, -1.0, b)); >> but is just as same as >> PetscCall(VecAXPY(b, -1.0, u)); >> >> Thank you so much for that!!! >> > > 90 KSP unpreconditioned resid norm 9.749157899195e+05 true resid norm > 9.749157899195e+05 ||r(i)||/||b|| 1.340218569960e+01 > 91 KSP unpreconditioned resid norm 1.073123446417e+06 true resid norm > 1.073123446417e+06 ||r(i)||/||b|| 1.475224820050e+01 > 92 KSP unpreconditioned resid norm 1.170251286554e+06 true resid norm > 1.170251286554e+06 ||r(i)||/||b|| 1.608746644557e+01 > 93 KSP unpreconditioned resid norm 1.264719067990e+06 true resid norm > 1.264719067990e+06 ||r(i)||/||b|| 1.738611681365e+01 > 94 KSP unpreconditioned resid norm 1.329446257320e+06 true resid norm > 1.329446257320e+06 ||r(i)||/||b|| 1.827592270272e+01 > 95 KSP unpreconditioned resid norm 1.365944956372e+06 true resid norm > 1.365944956372e+06 ||r(i)||/||b|| 1.877767100504e+01 > 96 KSP unpreconditioned resid norm 1.369513563400e+06 true resid norm > 1.369513563400e+06 ||r(i)||/||b|| 1.882672871297e+01 > 97 KSP unpreconditioned resid norm 1.364905651654e+06 true resid norm > 1.364905651654e+06 ||r(i)||/||b|| 1.876338366353e+01 > 98 KSP unpreconditioned resid norm 1.352584030803e+06 true resid norm > 1.352584030803e+06 ||r(i)||/||b|| 1.859399810996e+01 > 99 KSP unpreconditioned resid norm 1.330589478009e+06 true resid norm > 1.330589478009e+06 ||r(i)||/||b|| 1.829163857903e+01 > 100 KSP unpreconditioned resid norm 1.312782529439e+06 true resid norm > 1.312782529439e+06 ||r(i)||/||b|| 1.804684612214e+01 > Linear solve did not converge due to DIVERGED_ITS iterations 100 > KSPSolve Time: 7579.364000 ms > norm_delta 1312782.529439, norm_b 72743.044439 > Norm of error 18.0468 iterations 100 > Residue of error 1.31278e+06 iterations 100 > > I ran with > > -pc_type jacobi -ksp_max_it 100 > > because GAMG takes a long time to setup on my laptop. Those numbers match > exactly. > > THanks, > > Matt > > I'm such a beginner T_T >> ---- Replied Message ---- >> From Matthew Knepley >> Date 7/2/2023 20:10 >> To ??? >> Cc PETSc >> Subject Re: [petsc-users] Question about residue norm in PETSc >> On Sun, Jul 2, 2023 at 8:05?AM Matthew Knepley wrote: >> >>> On Sun, Jul 2, 2023 at 7:53?AM ??? wrote: >>> >>>> Thanks for your reply! >>>> So sorry that I made a mistake in the description. >>>> I set the tolerances by: >>>> PetscCall(KSPSetTolerances(ksp, 1e-12, DBL_MIN, PETSC_DEFAULT, >>>> PETSC_DEFAULT)); >>>> and got (by passing `-ksp_norm_type unpreconditioned >>>> -ksp_monitor_true_residual`) >>>> 74 KSP unpreconditioned resid norm 7.256655641876e-08 true resid norm >>>> 7.256655641876e-08 ||r(i)||/||b|| 9.975738158726e-13 >>>> I'm wondering why the ` ||r(i)||/||b||` is different with mine which >>>> calculated by: >>>> ``` >>>> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) (which >>>> is 72743.044439) >>>> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >>>> PetscCall(MatMult(A, x, u)); >>>> PetscCall(VecAXPY(b, -1.0, u)); >>>> PetscCall(VecNorm(b, NORM_2, &norm_delta)); // (which is 0.039608) >>>> ``` >>>> and (norm_delta) / (norm_b) = 5.44496e-07 which is higher and >>>> different with the rtol I set (1e-12). >>>> Sorry again for the waste of your time. I would really appreciated if >>>> you could help me again! >>>> >>> >>> 1) 7.256655641876e-08 / 72743.044439 = 9.975738158726e-13 so >>> ||r_i||/||b|| is correct in the output >>> >>> 2) You are asking why you calculate a different residual? I will have to >>> run your code. >>> >> >> I built your code, but you did not send the matrix and rhs. >> >> I suggest using >> >> PetscCall(VecAXPY(u, -1.0, b)); >> >> instead so that you do not change b, and keep the residual in u. >> >> Thanks, >> >> Matt >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best regards! >>>> Hemeng Wang >>>> >>>> >>>> ---- Replied Message ---- >>>> From Matthew Knepley >>>> Date 7/2/2023 18:51 >>>> To ??? >>>> Cc petsc-users at mcs.anl.gov >>>> >>>> Subject Re: [petsc-users] Question about residue norm in PETSc >>>> On Sun, Jul 2, 2023 at 2:24?AM ??? wrote: >>>> >>>>> Dear PETSc Team, >>>>> >>>>> Sorry to bother! My name is Hemeng Wang, and I am currently learning >>>>> the use of PETSc software package. I am confused while calculating the norm >>>>> of residue. >>>>> >>>>> I calculated residue norm by myself with: >>>>> ``` >>>>> PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) >>>>> >>>>> PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) >>>>> PetscCall(MatMult(A, x, u)); >>>>> PetscCall(VecAXPY(b, -1.0, u)); >>>>> PetscCall(VecNorm(b, NORM_2, &norm_delta)); >>>>> ``` >>>>> and check the (norm_delta) / (norm_b). It seems not the `atol` which >>>>> set by `KSPSetTolerances()`. >>>>> (I set atol as 1e-12, but got 5e-7 finally) >>>>> (options: -ksp_type cg -pc_type gamg -ksp_converged_reason >>>>> -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) >>>>> >>>> >>>> If you are using the default convergence test, there is an absolute >>>> tolerance (atol) _and_ a relative tolerance (rtol). It seems likely you hit >>>> the relative tolerance. You can check this using >>>> >>>> -ksp_converged_reason >>>> >>>> You could make rtol really small if you want to just see the atol >>>> >>>> -ksp_rtol 1e-20 >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> I also check the soure code of `KSPSolve_CG` in >>>>> `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the >>>>> difference. >>>>> >>>>> I will really really appreciated if someone can explain the >>>>> calculation of `resid norm` in petsc. And where is my mistake. >>>>> >>>>> To provide you with more context, here are the source code about my >>>>> implementation. And the output of my test. >>>>> >>>>> main.c >>>>> Main code of my program >>>>> >>>>> mmio.h mmloader.h >>>>> Headers for matrix read >>>>> >>>>> Makefile >>>>> For compiling, same as sample provided >>>>> >>>>> task.sh >>>>> A script for running program in `slurm` >>>>> >>>>> slurm-4803840.out >>>>> Output of my test >>>>> >>>>> Thank you very much for your time and attention. I greatly appreciate >>>>> your support and look forward to hearing from you soon. >>>>> Best regards, >>>>> Hemeng Wang >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 2 11:58:50 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 2 Jul 2023 12:58:50 -0400 Subject: [petsc-users] PCMG with PCREDISTRIBUTE In-Reply-To: References: <82DBCDB1-99C2-40A4-9741-D348AC5D5B3A@petsc.dev> <044DFE3A-95D6-48AA-B6DA-FBB228975597@unipv.it> <4347EFD4-D04E-4DEB-8313-B313AF5F4E02@unipv.it> <66740C62-780A-41C5-9228-1B91FE8D4115@petsc.dev> <7FDF0ACC-CBF1-427D-B1F2-425D8877DB9D@petsc.dev> Message-ID: <42623281-600D-4D09-9553-6A4927640266@petsc.dev> Are you sure the NN is correct? I cannot see how you set that and know that it exactly matches the way PCREDISTRIBUTE selects rows? I suggest making a tiny problem with artificial matrix values that you select to "slice" of parts of the grid, so you can see exactly on the grid that the selected rows and columns are correct as you expect. > On Jul 2, 2023, at 2:16 AM, Carl-Johan Thore wrote: > > Hi, > > I tried your suggestion > > ?1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good as on the full system.? > > from below and got the size of the interpolation matrix correct. But the convergence doesn?t seem right. In the attached .txt-file the code without redistribute convergences in 8 > FGMRES iterations whereas with redistribute it takes 25 (I?ve tested this on various meshes and the redistribute-code consistently performs much worse in terms of number of iterations). The code without redistribute is very well tested and always performs very well, so I?m fairly certain the error is in my new code. > > Would you be willing to take a quick look at the attached code snippet to see if I?m doing some obvious mistake? > > Kind regards, > Carl-Johan > > From: Barry Smith > > Sent: Friday, June 30, 2023 5:21 PM > To: Matthew Knepley > > Cc: Carl-Johan Thore >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > > > On Jun 30, 2023, at 10:22 AM, Matthew Knepley > wrote: > > On Fri, Jun 30, 2023 at 10:16?AM Carl-Johan Thore via petsc-users > wrote: > Thanks for the quick reply and the suggestions! > > ? ? you should first check that the PCMG works quite well ? > > Yes, the PCMG works very well for the full system. > > ?I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1.? > > Yes, it?s slightly different so problem 2 should be solved. > > It looked somewhat complicated to get PCMG to work with redistribute, so I?ll try with PCGAMG first > (it ran immediately with redistribute, but was slower than PCMG on my, very small, test problem. I?ll try > to tune the settings). > > A related question: I?m here using a DMDA for a structured grid but I?m locking so many DOFs that for many of the elements > all DOFs are locked. In such a case could it make sense to switch/convert the DMDA to a DMPlex containing only those > elements that actually have DOFs? > > Possibly, but if you are doing FD, then there is built-in topology in DMDA that is not present in Plex, so > finding the neighbors in the right order is harder (possible, but harder, we address this in some new work that is not yet merged). There is also structured adaptive support with DMForest, but this also does not preserve the stencil. > > The efficiency of active set VI solvers in PETSc demonstrates to me that solving reduced systems can be done efficiently with geometric multigrid using a structured grid so I would not suggest giving up on what you started. > > You can do it in two steps > > 1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good as on the full system. > > 2) the only problem with 1 is it is likely to be poorly load balanced (but you can make some runs to see how imbalanced it is, that will depend exactly on what parts are locked and what MPI processes they are on). So if it is poorly balanced then you would need to get out of redistribute.c a mapping for each kept dof to what MPI rank it is moved to and use that to move the entries in the reduced interpolation you have created. > > If you do succeed it would actually be useful code that we could add to PCREDISTRIBUTE for more general use by others. > > Barry > > > > > > Thanks, > > Matt > > From: Barry Smith > > Sent: Friday, June 30, 2023 3:57 PM > To: Carl-Johan Thore > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > Oh, I forgot to mention you should first check that the PCMG works quite well for the full system (without the PCREDISTRIBUTE); the convergence > on the redistributed system (assuming you did all the work to get PCMG to work for you) should be very similar to (but not measurably better) than the convergence on the full system. > > > > > On Jun 30, 2023, at 9:17 AM, Barry Smith > wrote: > > > ex42.c provides directly the interpolation/restriction needed to move between levels in the loop > > for (k = 1; k < nlevels; k++) { > PetscCall(DMCreateInterpolation(da_list[k - 1], da_list[k], &R, NULL)); > PetscCall(PCMGSetInterpolation(pc, k, R)); > PetscCall(MatDestroy(&R)); > } > > The more standard alternative to this is to call KSPSetDM() and have the PCMG setup use the DM > to construct the interpolations (I don't know why ex42.c does this construction itself instead of having the KSPSetDM() process handle it but that doesn't matter). The end result is the same in both cases. > > Since PCREDISTRIBUTE builds its own new matrix (by using only certain rows and columns of the original matrix) the original interpolation > cannot be used for two reasons > > 1) (since it is for the full system) It is for the wrong problem. > > 2) In addition, if you ran with ex42.c the inner KSP does not have access to the interpolation that was constructed so you could not get PCMG to to work as indicated below. > > I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1. > > So the short answer is that there is no "canned" way to use the PCMG process trivially with PCDISTRIBUTE. > > To do what you want requires two additional steps > > 1) after you construct the full interpolation matrix (by using the DM) you need to remove the rows associated with the dof that have been removed by the "locked" variables (and the columns that are associated with coarse grid points that live on the removed points) so that the interpolation is the correct "size" for the smaller problem > > 2) since PCREDISTRIBUTE actually moves dof of freedom between MPI processes for load balancing after it has removed the locked variables you would need to do the exact same movement for the rows of the interpolation matrix that you have constructed (after you have removed the "locked" rows of the interpolation. > > Lots of bookkeeping to acheive 1 and 2 but conceptually simple. > > As an experiment you can try using PCGAMG on the redistributed matrix -redistribute_pc_type gamg to use algebraic multigrid just to see the time and convergence rates. Since GAMG creates its own interpolation based on the matrix and it will be built on the smaller redistributed matrix there will no issue with the wrong "sized" interpolation. Of course you have the overhead of algebraic multigrid and cannot take advantage of geometric multigrid. The GAMG approach may be satisfactory to your needs. > > If you are game for looking more closely at using redistribute with geometric multigrid and PETSc (which will require digging into PETSc source code and using internal information in the PETSc source code) you can start by looking at how we solve variational problems with SNES using reduced space active set methods. SNESVINEWTONRSLS /src/snes/impls/vi/rs/virs.c This code solves problem 1 see() it builds the entire interpolation and then pulls out the required non-locked part. Reduced space active set methods essentially lock the constrained dof and solve a smaller system without those dof at each iteration. > > But it does not solve problem 2. Moving the rows of the "smaller" interpolation to the correct MPI process based on where PCREDISTRIBUTE moved rows. To do this would requring looking at the PCREDISTRUBUTE code and extracting the information of where each row was moving and performing the process for the interpolation matrix. > src/ksp/pc/impls/redistribute/redistribute.c > > Barry > > > > > > > > > > > On Jun 30, 2023, at 8:21 AM, Carl-Johan Thore via petsc-users > wrote: > > Hi, > > I'm trying to run an iterative solver (FGMRES for example) with PCMG as preconditioner. The setup of PCMG > is done roughly as in ex42 of the PETSc-tutorials (https://petsc.org/main/src/ksp/ksp/tutorials/ex42.c.html). > Since I have many locked degrees-of-freedom I would like to use PCREDISTRIBUTE. However, this > results in (30039 is the number of DOFs after redistribute and 55539 the number before): > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Matrix dimensions of A and P are incompatible for MatProductType PtAP: A 30039x30039, P 55539x7803 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.19.0-238-g512d1ae6db4 GIT Date: 2023-04-24 16:37:00 +0200 > [0]PETSC ERROR: topopt on a arch-linux-c-opt Fri Jun 30 13:28:41 2023 > [0]PETSC ERROR: Configure options COPTFLAGS="-O3 -march=native" CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" CUDAOPTFLAGS=-O3 --with-cuda --with-cusp --with-debugging=0 --download-scalapack --download-hdf5 --download-zlib --download-mumps --download-parmetis --download-metis --download-ptscotch --download-hypre --download-spai > [0]PETSC ERROR: #1 MatProductSetFromOptions_Private() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:420 > [0]PETSC ERROR: #2 MatProductSetFromOptions() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:541 > [0]PETSC ERROR: #3 MatPtAP() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:9868 > [0]PETSC ERROR: #4 MatGalerkin() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:10899 > [0]PETSC ERROR: #5 PCSetUp_MG() at /mnt/c/mathware/petsc/src/ksp/pc/impls/mg/mg.c:1029 > [0]PETSC ERROR: #6 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #7 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #8 PCSetUp_Redistribute() at /mnt/c/mathware/petsc/src/ksp/pc/impls/redistribute/redistribute.c:327 > [0]PETSC ERROR: #9 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #10 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #11 KSPSolve_Private() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:824 > [0]PETSC ERROR: #12 KSPSolve() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:1070 > > It?s clear what happens I think, and it kind of make since not all levels are redistributed as they should (?). > Is it possible to use PCMG with PCREDISTRIBUTE in an easy way? > > Kind regards, > Carl-Johan > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > ?? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: code_snippet.cc Type: application/octet-stream Size: 4716 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pcmg_redsitribute_initial_attempts.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 2 12:02:21 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 2 Jul 2023 13:02:21 -0400 Subject: [petsc-users] Question about residue norm in PETSc In-Reply-To: References: Message-ID: Also look at https://petsc.org/release/manualpages/KSP/KSPSetPCSide/#kspsetpcside and https://petsc.org/release/manualpages/KSP/KSPSetNormType/#kspsetnormtype in PETSc different Krylov solvers have different default values for this. > On Jul 2, 2023, at 1:47 AM, ??? wrote: > > Dear PETSc Team, > > Sorry to bother! My name is Hemeng Wang, and I am currently learning the use of PETSc software package. I am confused while calculating the norm of residue. > > I calculated residue norm by myself with: > ``` > PetscCall(VecNorm(b, NORM_2, &norm_b)); // (main.c, line 74) > > PetscCall(VecDuplicate(b, &u)); // (main.c, line 105) > PetscCall(MatMult(A, x, u)); > PetscCall(VecAXPY(b, -1.0, u)); > PetscCall(VecNorm(b, NORM_2, &norm_delta)); > ``` > and check the (norm_delta) / (norm_b). It seems not the `atol` which set by `KSPSetTolerances()`. > (I set atol as 1e-12, but got 5e-7 finally) > (options: -ksp_type cg -pc_type gamg -ksp_converged_reason -ksp_norm_type unpreconditioned -ksp_monitor_true_residual) > > I also check the soure code of `KSPSolve_CG` in `petsc/src/ksp/ksp/impls/cg/cg.c`. And could not figure out where is the difference. > > I will really really appreciated if someone can explain the calculation of `resid norm` in petsc. And where is my mistake. > > To provide you with more context, here are the source code about my implementation. And the output of my test. > > main.c > Main code of my program > > mmio.h mmloader.h > Headers for matrix read > > Makefile > For compiling, same as sample provided > > task.sh > A script for running program in `slurm` > > slurm-4803840.out > Output of my test > > Thank you very much for your time and attention. I greatly appreciate your support and look forward to hearing from you soon. > Best regards, > Hemeng Wang > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Sun Jul 2 22:24:36 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Mon, 3 Jul 2023 11:24:36 +0800 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: Hi! Good advice! I set value with MatSetValues() API, which sets one part of a row at a time(I use a kind of tiling technology so I cannot get all values of a row at a time). I tested the number of malloc in these three cases. The number of mallocs is decreasing with the increase of processors, and all these are very large(the matrix is 283234*283234, as can see in the following). This should be due to the unqualified preallocation. I use a rough preallocation, that every processor counts the number of nonzero elements for the first 10 rows, and uses the largest one to preallocate memory for all local rows. It seems that not work well. number_of_processors number_of_max_mallocs_among_all_processors 64 20000 128 17000 256 11000 I change my way to preallocate. I evenly take 100 rows in every local matrix and take the largest one to preallocate memory for all local rows. Now the assemble time is reduced to a very small time. number_of_processors number_of_max_mallocs_among_all_processors 64 3000 128 700 256 500 Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 3 0 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 3 0 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 3 0 So the reason "why assemble time is smaller with the increasing number of processors " may be because more processors divide the malloc job so that total time is reduced? If so, I still have some questions: 1. If preallocation is not accurate, will the performance of the assembly be affected? I mean, when processors receive the elements that should be stored in their local by MPI, then will the new mallocs happen at this time point? 2. I can not give an accurate preallocation for the large cost, so is there any better way to preallocate for my situation? Barry Smith ?2023?7?2??? 00:16??? > > I see no reason not to trust the times below, they seem reasonable. You > get more than 2 times speed from 64 to 128 and then about 1.38 from 128 to > 256. > > The total amount of data moved (number of messages moved times average > length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 > to 2.3000e+09. A pretty moderate amount of data increase, but note that > each time you double the number of ranks, you also increase substantially > the network's hardware to move data, so one would hope for a good speed up. > > Also, the load balance is very good, near 1. Often with assembly, we > see very out-of-balance, and it is difficult to get good speedup when the > balance is really off. > > It looks like over 90% of the entire run time is coming from setting > and assembling the values? Also the setting values time dominates assembly > time more with more ranks. Are you setting a single value at a time or a > collection of them? How big are the vectors? > > Run all three cases with -info :vec to see some information about how > many mallocs where move to hold the stashed vector entries. > > > > > On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: > > > > Hi, > Thanks for your reply. I try to use PetscLogEvent(), and the result > shows same conclusion. > What I have done is : > ---------------- > PetscLogEvent Mat_assemble_event, Mat_setvalue_event, > Mat_setAsse_event; > PetscClassId classid; > PetscLogDouble user_event_flops; > PetscClassIdRegister("Test assemble and set value", &classid); > PetscLogEventRegister("Test only assemble", classid, > &Mat_assemble_event); > PetscLogEventRegister("Test only set values", classid, > &Mat_setvalue_event); > PetscLogEventRegister("Test both assemble and set values", classid, > &Mat_setAsse_event); > PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); > PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); > ...compute elements and use MatSetValues. No call for assembly > PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); > > PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); > PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); > ---------------- > > And the output as follows. By the way, dose petsc recorde all time > between PetscLogEventBegin and PetscLogEventEnd? or just test the time of > petsc API? > > > It is all of the time. > > ---------------- > Event Count Time (sec) Flop > --- Global --- --- Stage ---- Total > Max Ratio *Max* Ratio Max Ratio Mess AvgLen > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 > 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 > 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 > 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 > 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 > 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 > > 64: > only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 2.8e+05 > 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 > only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 > 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 > > 128: > only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 > 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 > only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 > Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 > 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 > > 256: > only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 > 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 > only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 > Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 > 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 > > > > Runfeng > > Barry Smith ?2023?6?30??? 23:35??? > >> >> You cannot look just at the VecAssemblyEnd() time, that will very >> likely give the wrong impression of the total time it takes to put the >> values in. >> >> You need to register a new Event and put a PetscLogEvent() just before >> you start generating the vector entries and calling VecSetValues() and put >> the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way >> to get an accurate accounting of the time. >> >> Barry >> >> >> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin wrote: >> > >> > Hello! >> > >> > When I use PETSc build a sbaij matrix, I find a strange thing. When I >> increase the number of processors, the assemble time become smaller. All >> these are totally same matrix. The assemble time mainly arouse from message >> passing, which because I use dynamic workload that it is random for which >> elements are computed by which processor. >> > But from instinct, if use more processors, then more possible that the >> processor computes elements storing in other processors. But from the >> output of log_view, It seems when use more processors, the processors >> compute more elements storing in its local(infer from that, with more >> processors, less total amount of passed messages). >> > >> > What could cause this happened? Thank you! >> > >> > >> > Following is the output of log_view for 64\128\256 processors. Every >> row is time profiler of VecAssemblyEnd. >> > >> > >> ------------------------------------------------------------------------------------------------------------------------ >> > processors Count Time (sec) >> Flop >> --- Global --- --- Stage >> ---- Total >> > Max Ratio Max >> Ratio Max Ratio Mess AvgLen Reduct >> %T %F %M %L %R %T %F %M %L %R Mflop/s >> > 64 1 1.0 2.3775e+02 1.0 >> 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >> 52 0 1 1 2 52 0 1 1 2 >> 0 >> > 128 1 1.0 6.9945e+01 1.0 >> 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >> 30 0 1 1 2 30 0 1 1 2 >> 0 >> > 256 1 1.0 1.7445e+01 1.0 >> 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >> 10 0 1 1 2 10 0 1 1 2 >> 0 >> > >> > Runfeng Jin >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 2 23:18:58 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 3 Jul 2023 00:18:58 -0400 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: The main branch of PETSc now supports filling sparse matrices without providing any preallocation information. You can give it a try. Use your current fastest code but just remove ALL the preallocation calls. I would be interested in what kind of performance you get compared to your best current performance. Barry > On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: > > Hi! Good advice! > I set value with MatSetValues() API, which sets one part of a row at a time(I use a kind of tiling technology so I cannot get all values of a row at a time). > I tested the number of malloc in these three cases. The number of mallocs is decreasing with the increase of processors, and all these are very large(the matrix is 283234*283234, as can see in the following). This should be due to the unqualified preallocation. I use a rough preallocation, that every processor counts the number of nonzero elements for the first 10 rows, and uses the largest one to preallocate memory for all local rows. It seems that not work well. > number_of_processors number_of_max_mallocs_among_all_processors > 64 20000 > 128 17000 > 256 11000 > I change my way to preallocate. I evenly take 100 rows in every local matrix and take the largest one to preallocate memory for all local rows. Now the assemble time is reduced to a very small time. > number_of_processors number_of_max_mallocs_among_all_processors > 64 3000 > 128 700 > 256 500 > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 3 0 > 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 3 0 > 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 3 0 > So the reason "why assemble time is smaller with the increasing number of processors " may be because more processors divide the malloc job so that total time is reduced? > If so, I still have some questions: > 1. If preallocation is not accurate, will the performance of the assembly be affected? I mean, when processors receive the elements that should be stored in their local by MPI, then will the new mallocs happen at this time point? > 2. I can not give an accurate preallocation for the large cost, so is there any better way to preallocate for my situation? > > > Barry Smith > ?2023?7?2??? 00:16??? >> >> I see no reason not to trust the times below, they seem reasonable. You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 to 256. >> >> The total amount of data moved (number of messages moved times average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 to 2.3000e+09. A pretty moderate amount of data increase, but note that each time you double the number of ranks, you also increase substantially the network's hardware to move data, so one would hope for a good speed up. >> >> Also, the load balance is very good, near 1. Often with assembly, we see very out-of-balance, and it is difficult to get good speedup when the balance is really off. >> >> It looks like over 90% of the entire run time is coming from setting and assembling the values? Also the setting values time dominates assembly time more with more ranks. Are you setting a single value at a time or a collection of them? How big are the vectors? >> >> Run all three cases with -info :vec to see some information about how many mallocs where move to hold the stashed vector entries. >> >> >> >> >>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin > wrote: >>> >>> >>> >>> Hi, >>> Thanks for your reply. I try to use PetscLogEvent(), and the result shows same conclusion. >>> What I have done is : >>> ---------------- >>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, Mat_setAsse_event; >>> PetscClassId classid; >>> PetscLogDouble user_event_flops; >>> PetscClassIdRegister("Test assemble and set value", &classid); >>> PetscLogEventRegister("Test only assemble", classid, &Mat_assemble_event); >>> PetscLogEventRegister("Test only set values", classid, &Mat_setvalue_event); >>> PetscLogEventRegister("Test both assemble and set values", classid, &Mat_setAsse_event); >>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>> ...compute elements and use MatSetValues. No call for assembly >>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>> >>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>> ---------------- >>> >>> And the output as follows. By the way, dose petsc recorde all time between PetscLogEventBegin and PetscLogEventEnd? or just test the time of petsc API? >> >> It is all of the time. >> >>> ---------------- >>> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >>> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> 64new 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>> 128new 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>> 256new 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>> >>> 64: >>> only assemble 1 1.0 2.6596e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>> only setvalues 1 1.0 1.9987e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>> Test both 1 1.0 4.6580e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>> >>> 128: >>> only assemble 1 1.0 6.9718e+01 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>> only setvalues 1 1.0 1.4438e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>> Test both 1 1.0 2.1417e+02 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>> >>> 256: >>> only assemble 1 1.0 1.7482e+01 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>> only setvalues 1 1.0 1.3717e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>> Test both 1 1.0 1.5475e+02 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>> >>> >>> >>> Runfeng >>> >>> Barry Smith > ?2023?6?30??? 23:35??? >>>> >>>> You cannot look just at the VecAssemblyEnd() time, that will very likely give the wrong impression of the total time it takes to put the values in. >>>> >>>> You need to register a new Event and put a PetscLogEvent() just before you start generating the vector entries and calling VecSetValues() and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way to get an accurate accounting of the time. >>>> >>>> Barry >>>> >>>> >>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin > wrote: >>>> > >>>> > Hello! >>>> > >>>> > When I use PETSc build a sbaij matrix, I find a strange thing. When I increase the number of processors, the assemble time become smaller. All these are totally same matrix. The assemble time mainly arouse from message passing, which because I use dynamic workload that it is random for which elements are computed by which processor. >>>> > But from instinct, if use more processors, then more possible that the processor computes elements storing in other processors. But from the output of log_view, It seems when use more processors, the processors compute more elements storing in its local(infer from that, with more processors, less total amount of passed messages). >>>> > >>>> > What could cause this happened? Thank you! >>>> > >>>> > >>>> > Following is the output of log_view for 64\128\256 processors. Every row is time profiler of VecAssemblyEnd. >>>> > >>>> > ------------------------------------------------------------------------------------------------------------------------ >>>> > processors Count Time (sec) Flop --- Global --- --- Stage ---- Total >>>> > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>> > 64 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>> > 128 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>> > 256 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>> > >>>> > Runfeng Jin >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Mon Jul 3 08:55:44 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Mon, 3 Jul 2023 21:55:44 +0800 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: Hi, impressive performance! I use the newest version of petsc(release branch), and it almost deletes all assembly and stash time in large processors (assembly time 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming cost, it really incredible. The order code has a regular arrangement of the number of nonzero-elements across rows, so I can have a good rough preallocation. And from the data, dedicatedly arrange data and roughly acquiring the max number of non-zero elements in rows can have a better performance than the new version without preallocation. However, in reality, I will use the newer version without preallocation for:1)less effort in programming and also nearly the same good performance 2) good memory usage(I see no unneeded memory after assembly) 3) dedicated preallocation is usually not very easy and cause extra time cost. Maybe it will be better that leave some space for the user to do a slight direction for the preallocation and thus acquire better performance. But have no idea how to direct it. And I am very curious about how petsc achieves this. How can it not know anything but achieve so good performance, and no wasted memory? May you have an explanation about this? assemble time: version\processors 4 8 16 32 64 128 256 old 14677s 4694s 1124s 572s 38s 8s 2s new 50s 28s 15s 7.8s 4s 2s 0.4s older 27s 24s 19s 12s 14s - - stash time(max among all processors): version\processors 4 8 16 32 64 128 256 old 3145s 2554s 673s 329s 201s 142s 138s new 2s 1s ~0s ~0s ~0s ~0s ~0s older 10s 73s 18s 5s 1s - - old: my poor preallocation new: newest version of petsc that do not preallocation older: the best preallocation version of my code. Runfeng Barry Smith ?2023?7?3??? 12:19??? > > The main branch of PETSc now supports filling sparse matrices without > providing any preallocation information. > > You can give it a try. Use your current fastest code but just remove > ALL the preallocation calls. I would be interested in what kind of > performance you get compared to your best current performance. > > Barry > > > > On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: > > Hi! Good advice! > I set value with MatSetValues() API, which sets one part of a row at a > time(I use a kind of tiling technology so I cannot get all values of a row > at a time). > I tested the number of malloc in these three cases. The number of > mallocs is decreasing with the increase of processors, and all these are > very large(the matrix is 283234*283234, as can see in the following). This > should be due to the unqualified preallocation. I use a rough > preallocation, that every processor counts the number of nonzero elements > for the first 10 rows, and uses the largest one to preallocate memory for > all local rows. It seems that not work well. > > number_of_processors number_of_max_mallocs_among_all_processors > 64 20000 > 128 17000 > 256 11000 > > I change my way to preallocate. I evenly take 100 rows in every local > matrix and take the largest one to preallocate memory for all local rows. > Now the assemble time is reduced to a very small time. > > number_of_processors number_of_max_mallocs_among_all_processors > 64 3000 > 128 700 > 256 500 > > Event Count Time (sec) Flop > --- Global --- > --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess > AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 7.1e+03 > 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 3 > 0 > > 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 2.6e+04 > 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 3 > 0 > 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 1.0e+05 > 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 3 > 0 > > So the reason "why assemble time is smaller with the increasing number of > processors " may be because more processors divide the malloc job so that > total time is reduced? > If so, I still have some questions: > 1. If preallocation is not accurate, will the performance of the > assembly be affected? I mean, when processors receive the elements that > should be stored in their local by MPI, then will the new mallocs happen > at this time point? > 2. I can not give an accurate preallocation for the large cost, so is > there any better way to preallocate for my situation? > > > > Barry Smith ?2023?7?2??? 00:16??? > >> >> I see no reason not to trust the times below, they seem reasonable. >> You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 >> to 256. >> >> The total amount of data moved (number of messages moved times average >> length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 >> to 2.3000e+09. A pretty moderate amount of data increase, but note that >> each time you double the number of ranks, you also increase substantially >> the network's hardware to move data, so one would hope for a good speed up. >> >> Also, the load balance is very good, near 1. Often with assembly, we >> see very out-of-balance, and it is difficult to get good speedup when the >> balance is really off. >> >> It looks like over 90% of the entire run time is coming from setting >> and assembling the values? Also the setting values time dominates assembly >> time more with more ranks. Are you setting a single value at a time or a >> collection of them? How big are the vectors? >> >> Run all three cases with -info :vec to see some information about how >> many mallocs where move to hold the stashed vector entries. >> >> >> >> >> On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: >> >> >> >> Hi, >> Thanks for your reply. I try to use PetscLogEvent(), and the result >> shows same conclusion. >> What I have done is : >> ---------------- >> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, >> Mat_setAsse_event; >> PetscClassId classid; >> PetscLogDouble user_event_flops; >> PetscClassIdRegister("Test assemble and set value", &classid); >> PetscLogEventRegister("Test only assemble", classid, >> &Mat_assemble_event); >> PetscLogEventRegister("Test only set values", classid, >> &Mat_setvalue_event); >> PetscLogEventRegister("Test both assemble and set values", classid, >> &Mat_setAsse_event); >> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >> ...compute elements and use MatSetValues. No call for assembly >> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >> >> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >> ---------------- >> >> And the output as follows. By the way, dose petsc recorde all time >> between PetscLogEventBegin and PetscLogEventEnd? or just test the time of >> petsc API? >> >> >> It is all of the time. >> >> ---------------- >> Event Count Time (sec) Flop >> --- Global --- --- Stage ---- Total >> Max Ratio *Max* Ratio Max Ratio Mess >> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 >> 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >> 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 >> 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >> 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 >> 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >> >> 64: >> only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 2.8e+05 >> 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >> only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >> Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 >> 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >> >> 128: >> only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 >> 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >> only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >> Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 >> 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >> >> 256: >> only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 >> 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >> only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >> Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 >> 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >> >> >> >> Runfeng >> >> Barry Smith ?2023?6?30??? 23:35??? >> >>> >>> You cannot look just at the VecAssemblyEnd() time, that will very >>> likely give the wrong impression of the total time it takes to put the >>> values in. >>> >>> You need to register a new Event and put a PetscLogEvent() just >>> before you start generating the vector entries and calling VecSetValues() >>> and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the >>> only way to get an accurate accounting of the time. >>> >>> Barry >>> >>> >>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin wrote: >>> > >>> > Hello! >>> > >>> > When I use PETSc build a sbaij matrix, I find a strange thing. When I >>> increase the number of processors, the assemble time become smaller. All >>> these are totally same matrix. The assemble time mainly arouse from message >>> passing, which because I use dynamic workload that it is random for which >>> elements are computed by which processor. >>> > But from instinct, if use more processors, then more possible that the >>> processor computes elements storing in other processors. But from the >>> output of log_view, It seems when use more processors, the processors >>> compute more elements storing in its local(infer from that, with more >>> processors, less total amount of passed messages). >>> > >>> > What could cause this happened? Thank you! >>> > >>> > >>> > Following is the output of log_view for 64\128\256 processors. Every >>> row is time profiler of VecAssemblyEnd. >>> > >>> > >>> ------------------------------------------------------------------------------------------------------------------------ >>> > processors Count Time (sec) >>> Flop >>> --- Global --- --- Stage >>> ---- Total >>> > Max Ratio Max >>> Ratio Max Ratio Mess AvgLen Reduct >>> %T %F %M %L %R %T %F %M %L %R Mflop/s >>> > 64 1 1.0 2.3775e+02 >>> 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >>> 52 0 1 1 2 52 0 1 1 2 >>> 0 >>> > 128 1 1.0 6.9945e+01 1.0 >>> 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >>> 30 0 1 1 2 30 0 1 1 2 >>> 0 >>> > 256 1 1.0 1.7445e+01 >>> 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >>> 10 0 1 1 2 10 0 1 1 2 >>> 0 >>> > >>> > Runfeng Jin >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 3 09:03:45 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 Jul 2023 10:03:45 -0400 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: On Mon, Jul 3, 2023 at 9:56?AM Runfeng Jin wrote: > Hi, impressive performance! > I use the newest version of petsc(release branch), and it almost deletes > all assembly and stash time in large processors (assembly time > 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming > cost, it really incredible. > The order code has a regular arrangement of the number of > nonzero-elements across rows, so I can have a good rough preallocation. And > from the data, dedicatedly arrange data and roughly acquiring the max > number of non-zero elements in rows can have a better performance than the > new version without preallocation. However, in reality, I will use the > newer version without preallocation for:1)less effort in programming and > also nearly the same good performance 2) good memory usage(I see no > unneeded memory after assembly) 3) dedicated preallocation is usually not > very easy and cause extra time cost. > Maybe it will be better that leave some space for the user to do a > slight direction for the preallocation and thus acquire better performance. > But have no idea how to direct it. > And I am very curious about how petsc achieves this. How can it not > know anything but achieve so good performance, and no wasted memory? May > you have an explanation about this? > We use a hash table to store the nonzeros on the fly, and then convert to packed storage on assembly. Thanks, Matt > assemble time: > version\processors 4 8 16 32 > 64 128 256 > old 14677s 4694s 1124s 572s > 38s 8s 2s > new 50s 28s 15s > 7.8s 4s 2s 0.4s > older 27s 24s 19s > 12s 14s - - > stash time(max among all processors): > version\processors 4 8 16 32 > 64 128 256 > old 3145s 2554s 673s 329s > 201s 142s 138s > new 2s 1s ~0s > ~0s ~0s ~0s ~0s > older 10s 73s 18s 5s > 1s - - > old: my poor preallocation > new: newest version of petsc that do not preallocation > older: the best preallocation version of my code. > > > Runfeng > > Barry Smith ?2023?7?3??? 12:19??? > >> >> The main branch of PETSc now supports filling sparse matrices without >> providing any preallocation information. >> >> You can give it a try. Use your current fastest code but just remove >> ALL the preallocation calls. I would be interested in what kind of >> performance you get compared to your best current performance. >> >> Barry >> >> >> >> On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: >> >> Hi! Good advice! >> I set value with MatSetValues() API, which sets one part of a row at >> a time(I use a kind of tiling technology so I cannot get all values of a >> row at a time). >> I tested the number of malloc in these three cases. The number of >> mallocs is decreasing with the increase of processors, and all these are >> very large(the matrix is 283234*283234, as can see in the following). This >> should be due to the unqualified preallocation. I use a rough >> preallocation, that every processor counts the number of nonzero elements >> for the first 10 rows, and uses the largest one to preallocate memory for >> all local rows. It seems that not work well. >> >> number_of_processors number_of_max_mallocs_among_all_processors >> 64 20000 >> 128 17000 >> 256 11000 >> >> I change my way to preallocate. I evenly take 100 rows in every local >> matrix and take the largest one to preallocate memory for all local rows. >> Now the assemble time is reduced to a very small time. >> >> number_of_processors number_of_max_mallocs_among_all_processors >> 64 3000 >> 128 700 >> 256 500 >> >> Event Count Time (sec) Flop >> --- Global --- >> --- Stage ---- Total >> Max Ratio Max Ratio Max Ratio Mess >> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 >> 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 >> 3 0 >> >> 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 2.6e+04 >> 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 3 >> 0 >> 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 1.0e+05 >> 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 3 >> 0 >> >> So the reason "why assemble time is smaller with the increasing number of >> processors " may be because more processors divide the malloc job so that >> total time is reduced? >> If so, I still have some questions: >> 1. If preallocation is not accurate, will the performance of the >> assembly be affected? I mean, when processors receive the elements that >> should be stored in their local by MPI, then will the new mallocs happen >> at this time point? >> 2. I can not give an accurate preallocation for the large cost, so is >> there any better way to preallocate for my situation? >> >> >> >> Barry Smith ?2023?7?2??? 00:16??? >> >>> >>> I see no reason not to trust the times below, they seem reasonable. >>> You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 >>> to 256. >>> >>> The total amount of data moved (number of messages moved times >>> average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 >>> to 2.3000e+09. A pretty moderate amount of data increase, but note that >>> each time you double the number of ranks, you also increase substantially >>> the network's hardware to move data, so one would hope for a good speed up. >>> >>> Also, the load balance is very good, near 1. Often with assembly, we >>> see very out-of-balance, and it is difficult to get good speedup when the >>> balance is really off. >>> >>> It looks like over 90% of the entire run time is coming from setting >>> and assembling the values? Also the setting values time dominates assembly >>> time more with more ranks. Are you setting a single value at a time or a >>> collection of them? How big are the vectors? >>> >>> Run all three cases with -info :vec to see some information about how >>> many mallocs where move to hold the stashed vector entries. >>> >>> >>> >>> >>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: >>> >>> >>> >>> Hi, >>> Thanks for your reply. I try to use PetscLogEvent(), and the result >>> shows same conclusion. >>> What I have done is : >>> ---------------- >>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, >>> Mat_setAsse_event; >>> PetscClassId classid; >>> PetscLogDouble user_event_flops; >>> PetscClassIdRegister("Test assemble and set value", &classid); >>> PetscLogEventRegister("Test only assemble", classid, >>> &Mat_assemble_event); >>> PetscLogEventRegister("Test only set values", classid, >>> &Mat_setvalue_event); >>> PetscLogEventRegister("Test both assemble and set values", classid, >>> &Mat_setAsse_event); >>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>> ...compute elements and use MatSetValues. No call for assembly >>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>> >>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>> ---------------- >>> >>> And the output as follows. By the way, dose petsc recorde all time >>> between PetscLogEventBegin and PetscLogEventEnd? or just test the time of >>> petsc API? >>> >>> >>> It is all of the time. >>> >>> ---------------- >>> Event Count Time (sec) Flop >>> --- Global --- --- Stage ---- Total >>> Max Ratio *Max* Ratio Max Ratio Mess >>> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 >>> 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>> 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 >>> 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>> 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 >>> 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>> >>> 64: >>> only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 2.8e+05 >>> 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>> only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>> Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 >>> 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>> >>> 128: >>> only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 >>> 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>> only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>> Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 >>> 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>> >>> 256: >>> only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 >>> 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>> only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>> Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 >>> 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>> >>> >>> >>> Runfeng >>> >>> Barry Smith ?2023?6?30??? 23:35??? >>> >>>> >>>> You cannot look just at the VecAssemblyEnd() time, that will very >>>> likely give the wrong impression of the total time it takes to put the >>>> values in. >>>> >>>> You need to register a new Event and put a PetscLogEvent() just >>>> before you start generating the vector entries and calling VecSetValues() >>>> and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the >>>> only way to get an accurate accounting of the time. >>>> >>>> Barry >>>> >>>> >>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin >>>> wrote: >>>> > >>>> > Hello! >>>> > >>>> > When I use PETSc build a sbaij matrix, I find a strange thing. When I >>>> increase the number of processors, the assemble time become smaller. All >>>> these are totally same matrix. The assemble time mainly arouse from message >>>> passing, which because I use dynamic workload that it is random for which >>>> elements are computed by which processor. >>>> > But from instinct, if use more processors, then more possible that >>>> the processor computes elements storing in other processors. But from the >>>> output of log_view, It seems when use more processors, the processors >>>> compute more elements storing in its local(infer from that, with more >>>> processors, less total amount of passed messages). >>>> > >>>> > What could cause this happened? Thank you! >>>> > >>>> > >>>> > Following is the output of log_view for 64\128\256 processors. Every >>>> row is time profiler of VecAssemblyEnd. >>>> > >>>> > >>>> ------------------------------------------------------------------------------------------------------------------------ >>>> > processors Count Time (sec) >>>> Flop >>>> --- Global --- --- >>>> Stage ---- Total >>>> > Max Ratio Max >>>> Ratio Max Ratio Mess AvgLen Reduct >>>> %T %F %M %L %R %T %F %M %L %R Mflop/s >>>> > 64 1 1.0 2.3775e+02 >>>> 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >>>> 52 0 1 1 2 52 0 1 1 2 >>>> 0 >>>> > 128 1 1.0 6.9945e+01 >>>> 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >>>> 30 0 1 1 2 30 0 1 1 2 >>>> 0 >>>> > 256 1 1.0 1.7445e+01 >>>> 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >>>> 10 0 1 1 2 10 0 1 1 2 >>>> 0 >>>> > >>>> > Runfeng Jin >>>> >>>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Mon Jul 3 09:05:25 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Mon, 3 Jul 2023 22:05:25 +0800 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: Thank you for all your help! Runfeng Matthew Knepley ?2023?7?3??? 22:03??? > On Mon, Jul 3, 2023 at 9:56?AM Runfeng Jin wrote: > >> Hi, impressive performance! >> I use the newest version of petsc(release branch), and it almost >> deletes all assembly and stash time in large processors (assembly time >> 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming >> cost, it really incredible. >> The order code has a regular arrangement of the number of >> nonzero-elements across rows, so I can have a good rough preallocation. And >> from the data, dedicatedly arrange data and roughly acquiring the max >> number of non-zero elements in rows can have a better performance than the >> new version without preallocation. However, in reality, I will use the >> newer version without preallocation for:1)less effort in programming and >> also nearly the same good performance 2) good memory usage(I see no >> unneeded memory after assembly) 3) dedicated preallocation is usually not >> very easy and cause extra time cost. >> Maybe it will be better that leave some space for the user to do a >> slight direction for the preallocation and thus acquire better performance. >> But have no idea how to direct it. >> And I am very curious about how petsc achieves this. How can it not >> know anything but achieve so good performance, and no wasted memory? May >> you have an explanation about this? >> > > We use a hash table to store the nonzeros on the fly, and then convert to > packed storage on assembly. > > Thanks, > > Matt > > >> assemble time: >> version\processors 4 8 16 32 >> 64 128 256 >> old 14677s 4694s 1124s 572s >> 38s 8s 2s >> new 50s 28s 15s >> 7.8s 4s 2s 0.4s >> older 27s 24s 19s >> 12s 14s - - >> stash time(max among all processors): >> version\processors 4 8 16 32 >> 64 128 256 >> old 3145s 2554s 673s 329s >> 201s 142s 138s >> new 2s 1s ~0s >> ~0s ~0s ~0s ~0s >> older 10s 73s 18s >> 5s 1s - - >> old: my poor preallocation >> new: newest version of petsc that do not preallocation >> older: the best preallocation version of my code. >> >> >> Runfeng >> >> Barry Smith ?2023?7?3??? 12:19??? >> >>> >>> The main branch of PETSc now supports filling sparse matrices without >>> providing any preallocation information. >>> >>> You can give it a try. Use your current fastest code but just remove >>> ALL the preallocation calls. I would be interested in what kind of >>> performance you get compared to your best current performance. >>> >>> Barry >>> >>> >>> >>> On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: >>> >>> Hi! Good advice! >>> I set value with MatSetValues() API, which sets one part of a row at >>> a time(I use a kind of tiling technology so I cannot get all values of a >>> row at a time). >>> I tested the number of malloc in these three cases. The number of >>> mallocs is decreasing with the increase of processors, and all these are >>> very large(the matrix is 283234*283234, as can see in the following). This >>> should be due to the unqualified preallocation. I use a rough >>> preallocation, that every processor counts the number of nonzero elements >>> for the first 10 rows, and uses the largest one to preallocate memory for >>> all local rows. It seems that not work well. >>> >>> number_of_processors number_of_max_mallocs_among_all_processors >>> 64 20000 >>> 128 17000 >>> 256 11000 >>> >>> I change my way to preallocate. I evenly take 100 rows in every >>> local matrix and take the largest one to preallocate memory for all local >>> rows. Now the assemble time is reduced to a very small time. >>> >>> number_of_processors number_of_max_mallocs_among_all_processors >>> 64 3000 >>> 128 700 >>> 256 500 >>> >>> Event Count Time (sec) Flop >>> --- Global --- >>> --- Stage ---- Total >>> Max Ratio Max Ratio Max Ratio >>> Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R >>> Mflop/s >>> 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 >>> 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 >>> 3 0 >>> >>> 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 >>> 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 >>> 3 0 >>> 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 >>> 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 >>> 3 0 >>> >>> So the reason "why assemble time is smaller with the increasing number >>> of processors " may be because more processors divide the malloc job so >>> that total time is reduced? >>> If so, I still have some questions: >>> 1. If preallocation is not accurate, will the performance of the >>> assembly be affected? I mean, when processors receive the elements that >>> should be stored in their local by MPI, then will the new mallocs happen >>> at this time point? >>> 2. I can not give an accurate preallocation for the large cost, so >>> is there any better way to preallocate for my situation? >>> >>> >>> >>> Barry Smith ?2023?7?2??? 00:16??? >>> >>>> >>>> I see no reason not to trust the times below, they seem reasonable. >>>> You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 >>>> to 256. >>>> >>>> The total amount of data moved (number of messages moved times >>>> average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 >>>> to 2.3000e+09. A pretty moderate amount of data increase, but note that >>>> each time you double the number of ranks, you also increase substantially >>>> the network's hardware to move data, so one would hope for a good speed up. >>>> >>>> Also, the load balance is very good, near 1. Often with assembly, we >>>> see very out-of-balance, and it is difficult to get good speedup when the >>>> balance is really off. >>>> >>>> It looks like over 90% of the entire run time is coming from setting >>>> and assembling the values? Also the setting values time dominates assembly >>>> time more with more ranks. Are you setting a single value at a time or a >>>> collection of them? How big are the vectors? >>>> >>>> Run all three cases with -info :vec to see some information about >>>> how many mallocs where move to hold the stashed vector entries. >>>> >>>> >>>> >>>> >>>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: >>>> >>>> >>>> >>>> Hi, >>>> Thanks for your reply. I try to use PetscLogEvent(), and the result >>>> shows same conclusion. >>>> What I have done is : >>>> ---------------- >>>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, >>>> Mat_setAsse_event; >>>> PetscClassId classid; >>>> PetscLogDouble user_event_flops; >>>> PetscClassIdRegister("Test assemble and set value", &classid); >>>> PetscLogEventRegister("Test only assemble", classid, >>>> &Mat_assemble_event); >>>> PetscLogEventRegister("Test only set values", classid, >>>> &Mat_setvalue_event); >>>> PetscLogEventRegister("Test both assemble and set values", classid, >>>> &Mat_setAsse_event); >>>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>>> ...compute elements and use MatSetValues. No call for assembly >>>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>>> >>>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>>> ---------------- >>>> >>>> And the output as follows. By the way, dose petsc recorde all time >>>> between PetscLogEventBegin and PetscLogEventEnd? or just test the time of >>>> petsc API? >>>> >>>> >>>> It is all of the time. >>>> >>>> ---------------- >>>> Event Count Time (sec) Flop >>>> --- Global --- --- Stage ---- Total >>>> Max Ratio *Max* Ratio Max Ratio Mess >>>> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>> 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 >>>> 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>> 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 >>>> 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>> 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 >>>> 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>> >>>> 64: >>>> only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 >>>> 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>>> only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>>> Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 >>>> 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>>> >>>> 128: >>>> only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 >>>> 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>>> only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>>> Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 >>>> 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>>> >>>> 256: >>>> only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 >>>> 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>>> only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>>> Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 >>>> 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>>> >>>> >>>> >>>> Runfeng >>>> >>>> Barry Smith ?2023?6?30??? 23:35??? >>>> >>>>> >>>>> You cannot look just at the VecAssemblyEnd() time, that will very >>>>> likely give the wrong impression of the total time it takes to put the >>>>> values in. >>>>> >>>>> You need to register a new Event and put a PetscLogEvent() just >>>>> before you start generating the vector entries and calling VecSetValues() >>>>> and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the >>>>> only way to get an accurate accounting of the time. >>>>> >>>>> Barry >>>>> >>>>> >>>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin >>>>> wrote: >>>>> > >>>>> > Hello! >>>>> > >>>>> > When I use PETSc build a sbaij matrix, I find a strange thing. When >>>>> I increase the number of processors, the assemble time become smaller. All >>>>> these are totally same matrix. The assemble time mainly arouse from message >>>>> passing, which because I use dynamic workload that it is random for which >>>>> elements are computed by which processor. >>>>> > But from instinct, if use more processors, then more possible that >>>>> the processor computes elements storing in other processors. But from the >>>>> output of log_view, It seems when use more processors, the processors >>>>> compute more elements storing in its local(infer from that, with more >>>>> processors, less total amount of passed messages). >>>>> > >>>>> > What could cause this happened? Thank you! >>>>> > >>>>> > >>>>> > Following is the output of log_view for 64\128\256 processors. >>>>> Every row is time profiler of VecAssemblyEnd. >>>>> > >>>>> > >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> > processors Count Time (sec) >>>>> Flop >>>>> --- Global --- --- >>>>> Stage ---- Total >>>>> > Max Ratio Max >>>>> Ratio Max Ratio Mess AvgLen >>>>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>> > 64 1 1.0 2.3775e+02 >>>>> 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >>>>> 52 0 1 1 2 52 0 1 1 2 >>>>> 0 >>>>> > 128 1 1.0 6.9945e+01 >>>>> 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >>>>> 30 0 1 1 2 30 0 1 1 2 >>>>> 0 >>>>> > 256 1 1.0 1.7445e+01 >>>>> 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >>>>> 10 0 1 1 2 10 0 1 1 2 >>>>> 0 >>>>> > >>>>> > Runfeng Jin >>>>> >>>>> >>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Mon Jul 3 09:11:42 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Mon, 3 Jul 2023 22:11:42 +0800 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: Hi, > We use a hash table to store the nonzeros on the fly, and then convert to > packed storage on assembly. > Maybe can you tell me which file implements this function? Runfeng Runfeng Jin ?2023?7?3??? 22:05??? > Thank you for all your help! > > Runfeng > > Matthew Knepley ?2023?7?3??? 22:03??? > >> On Mon, Jul 3, 2023 at 9:56?AM Runfeng Jin wrote: >> >>> Hi, impressive performance! >>> I use the newest version of petsc(release branch), and it almost >>> deletes all assembly and stash time in large processors (assembly time >>> 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming >>> cost, it really incredible. >>> The order code has a regular arrangement of the number of >>> nonzero-elements across rows, so I can have a good rough preallocation. And >>> from the data, dedicatedly arrange data and roughly acquiring the max >>> number of non-zero elements in rows can have a better performance than the >>> new version without preallocation. However, in reality, I will use the >>> newer version without preallocation for:1)less effort in programming and >>> also nearly the same good performance 2) good memory usage(I see no >>> unneeded memory after assembly) 3) dedicated preallocation is usually not >>> very easy and cause extra time cost. >>> Maybe it will be better that leave some space for the user to do a >>> slight direction for the preallocation and thus acquire better performance. >>> But have no idea how to direct it. >>> And I am very curious about how petsc achieves this. How can it not >>> know anything but achieve so good performance, and no wasted memory? May >>> you have an explanation about this? >>> >> >> We use a hash table to store the nonzeros on the fly, and then convert to >> packed storage on assembly. >> >> Thanks, >> >> Matt >> >> >>> assemble time: >>> version\processors 4 8 16 32 >>> 64 128 256 >>> old 14677s 4694s 1124s 572s >>> 38s 8s 2s >>> new 50s 28s 15s >>> 7.8s 4s 2s 0.4s >>> older 27s 24s 19s >>> 12s 14s - - >>> stash time(max among all processors): >>> version\processors 4 8 16 32 >>> 64 128 256 >>> old 3145s 2554s 673s 329s >>> 201s 142s 138s >>> new 2s 1s ~0s >>> ~0s ~0s ~0s ~0s >>> older 10s 73s 18s >>> 5s 1s - - >>> old: my poor preallocation >>> new: newest version of petsc that do not preallocation >>> older: the best preallocation version of my code. >>> >>> >>> Runfeng >>> >>> Barry Smith ?2023?7?3??? 12:19??? >>> >>>> >>>> The main branch of PETSc now supports filling sparse matrices >>>> without providing any preallocation information. >>>> >>>> You can give it a try. Use your current fastest code but just remove >>>> ALL the preallocation calls. I would be interested in what kind of >>>> performance you get compared to your best current performance. >>>> >>>> Barry >>>> >>>> >>>> >>>> On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: >>>> >>>> Hi! Good advice! >>>> I set value with MatSetValues() API, which sets one part of a row >>>> at a time(I use a kind of tiling technology so I cannot get all values of a >>>> row at a time). >>>> I tested the number of malloc in these three cases. The number of >>>> mallocs is decreasing with the increase of processors, and all these are >>>> very large(the matrix is 283234*283234, as can see in the following). This >>>> should be due to the unqualified preallocation. I use a rough >>>> preallocation, that every processor counts the number of nonzero elements >>>> for the first 10 rows, and uses the largest one to preallocate memory for >>>> all local rows. It seems that not work well. >>>> >>>> number_of_processors number_of_max_mallocs_among_all_processors >>>> 64 20000 >>>> 128 17000 >>>> 256 11000 >>>> >>>> I change my way to preallocate. I evenly take 100 rows in every >>>> local matrix and take the largest one to preallocate memory for all local >>>> rows. Now the assemble time is reduced to a very small time. >>>> >>>> number_of_processors number_of_max_mallocs_among_all_processors >>>> 64 3000 >>>> 128 700 >>>> 256 500 >>>> >>>> Event Count Time (sec) Flop >>>> --- Global --- >>>> --- Stage ---- Total >>>> Max Ratio Max Ratio Max Ratio >>>> Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R >>>> Mflop/s >>>> 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 >>>> 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 >>>> 3 0 >>>> >>>> 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 >>>> 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 >>>> 3 0 >>>> 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 >>>> 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 >>>> 3 0 >>>> >>>> So the reason "why assemble time is smaller with the increasing number >>>> of processors " may be because more processors divide the malloc job so >>>> that total time is reduced? >>>> If so, I still have some questions: >>>> 1. If preallocation is not accurate, will the performance of the >>>> assembly be affected? I mean, when processors receive the elements that >>>> should be stored in their local by MPI, then will the new mallocs happen >>>> at this time point? >>>> 2. I can not give an accurate preallocation for the large cost, so >>>> is there any better way to preallocate for my situation? >>>> >>>> >>>> >>>> Barry Smith ?2023?7?2??? 00:16??? >>>> >>>>> >>>>> I see no reason not to trust the times below, they seem reasonable. >>>>> You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 >>>>> to 256. >>>>> >>>>> The total amount of data moved (number of messages moved times >>>>> average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 >>>>> to 2.3000e+09. A pretty moderate amount of data increase, but note that >>>>> each time you double the number of ranks, you also increase substantially >>>>> the network's hardware to move data, so one would hope for a good speed up. >>>>> >>>>> Also, the load balance is very good, near 1. Often with assembly, >>>>> we see very out-of-balance, and it is difficult to get good speedup when >>>>> the balance is really off. >>>>> >>>>> It looks like over 90% of the entire run time is coming from >>>>> setting and assembling the values? Also the setting values time dominates >>>>> assembly time more with more ranks. Are you setting a single value at a >>>>> time or a collection of them? How big are the vectors? >>>>> >>>>> Run all three cases with -info :vec to see some information about >>>>> how many mallocs where move to hold the stashed vector entries. >>>>> >>>>> >>>>> >>>>> >>>>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin wrote: >>>>> >>>>> >>>>> >>>>> Hi, >>>>> Thanks for your reply. I try to use PetscLogEvent(), and the >>>>> result shows same conclusion. >>>>> What I have done is : >>>>> ---------------- >>>>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, >>>>> Mat_setAsse_event; >>>>> PetscClassId classid; >>>>> PetscLogDouble user_event_flops; >>>>> PetscClassIdRegister("Test assemble and set value", &classid); >>>>> PetscLogEventRegister("Test only assemble", classid, >>>>> &Mat_assemble_event); >>>>> PetscLogEventRegister("Test only set values", classid, >>>>> &Mat_setvalue_event); >>>>> PetscLogEventRegister("Test both assemble and set values", >>>>> classid, &Mat_setAsse_event); >>>>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>>>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>>>> ...compute elements and use MatSetValues. No call for assembly >>>>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>>>> >>>>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>>>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>>>> ---------------- >>>>> >>>>> And the output as follows. By the way, dose petsc recorde all time >>>>> between PetscLogEventBegin and PetscLogEventEnd? or just test the time of >>>>> petsc API? >>>>> >>>>> >>>>> It is all of the time. >>>>> >>>>> ---------------- >>>>> Event Count Time (sec) Flop >>>>> --- Global --- --- Stage ---- Total >>>>> Max Ratio *Max* Ratio Max Ratio Mess >>>>> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>> 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 >>>>> 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>>> 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 >>>>> 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>>> 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 >>>>> 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>>> >>>>> 64: >>>>> only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 >>>>> 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>>>> only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>>>> Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 >>>>> 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>>>> >>>>> 128: >>>>> only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 >>>>> 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>>>> only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>>>> Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 >>>>> 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>>>> >>>>> 256: >>>>> only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 >>>>> 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>>>> only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>>> 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>>>> Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 >>>>> 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>>>> >>>>> >>>>> >>>>> Runfeng >>>>> >>>>> Barry Smith ?2023?6?30??? 23:35??? >>>>> >>>>>> >>>>>> You cannot look just at the VecAssemblyEnd() time, that will very >>>>>> likely give the wrong impression of the total time it takes to put the >>>>>> values in. >>>>>> >>>>>> You need to register a new Event and put a PetscLogEvent() just >>>>>> before you start generating the vector entries and calling VecSetValues() >>>>>> and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the >>>>>> only way to get an accurate accounting of the time. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin >>>>>> wrote: >>>>>> > >>>>>> > Hello! >>>>>> > >>>>>> > When I use PETSc build a sbaij matrix, I find a strange thing. When >>>>>> I increase the number of processors, the assemble time become smaller. All >>>>>> these are totally same matrix. The assemble time mainly arouse from message >>>>>> passing, which because I use dynamic workload that it is random for which >>>>>> elements are computed by which processor. >>>>>> > But from instinct, if use more processors, then more possible that >>>>>> the processor computes elements storing in other processors. But from the >>>>>> output of log_view, It seems when use more processors, the processors >>>>>> compute more elements storing in its local(infer from that, with more >>>>>> processors, less total amount of passed messages). >>>>>> > >>>>>> > What could cause this happened? Thank you! >>>>>> > >>>>>> > >>>>>> > Following is the output of log_view for 64\128\256 processors. >>>>>> Every row is time profiler of VecAssemblyEnd. >>>>>> > >>>>>> > >>>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>>> > processors Count Time (sec) >>>>>> Flop >>>>>> --- Global --- --- >>>>>> Stage ---- Total >>>>>> > Max Ratio Max >>>>>> Ratio Max Ratio Mess AvgLen >>>>>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>> > 64 1 1.0 2.3775e+02 >>>>>> 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >>>>>> 52 0 1 1 2 52 0 1 1 2 >>>>>> 0 >>>>>> > 128 1 1.0 6.9945e+01 >>>>>> 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >>>>>> 30 0 1 1 2 30 0 1 1 2 >>>>>> 0 >>>>>> > 256 1 1.0 1.7445e+01 >>>>>> 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >>>>>> 10 0 1 1 2 10 0 1 1 2 >>>>>> 0 >>>>>> > >>>>>> > Runfeng Jin >>>>>> >>>>>> >>>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 3 09:52:16 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 3 Jul 2023 10:52:16 -0400 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> Message-ID: <1D1523EB-62D4-451D-9961-9E6FAA05F1D2@petsc.dev> > On Jul 3, 2023, at 10:11 AM, Runfeng Jin wrote: > > Hi, >> We use a hash table to store the nonzeros on the fly, and then convert to packed storage on assembly. There is "extra memory" since the matrix entries are first stored in a hash and then converted into the regular CSR format, so for a short while, both copies are in memory. We use the amazing khash package, include/petsc/private/khash/khash.h, our code is scattered around a bit depending on the matrix format we will be forming. cd src/mat git grep "_Hash(" impls/aij/mpi/mpiaij.c:/* defines MatSetValues_MPI_Hash(), MatAssemblyBegin_MPI_Hash(), and MatAssemblyEnd_MPI_Hash() */ impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:/* defines MatSetValues_MPICUSPARSE_Hash() */ impls/aij/mpi/mpicusparse/mpiaijcusparse.cu: PetscCall(MatSetUp_MPI_Hash(A)); impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetValues_MPI_Hash(Mat A, PetscInt m, const PetscInt *rows, PetscInt n, const PetscInt *cols, const PetscScalar *values, InsertMode addv) impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatAssemblyBegin_MPI_Hash(Mat A, PETSC_UNUSED MatAssemblyType type) impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatAssemblyEnd_MPI_Hash(Mat A, MatAssemblyType type) impls/aij/mpi/mpihashmat.h: PetscCall(MatSetValues_MPI_Hash(A, 1, row + i, ncols, col + i, val + i, A->insertmode)); impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatDestroy_MPI_Hash(Mat A) impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatZeroEntries_MPI_Hash(PETSC_UNUSED Mat A) impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetRandom_MPI_Hash(Mat A, PETSC_UNUSED PetscRandom r) impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetUp_MPI_Hash(Mat A) impls/aij/seq/aij.c:/* defines MatSetValues_Seq_Hash(), MatAssemblyEnd_Seq_Hash(), MatSetUp_Seq_Hash() */ impls/aij/seq/seqhashmat.h:static PetscErrorCode MatAssemblyEnd_Seq_Hash(Mat A, MatAssemblyType type) impls/aij/seq/seqhashmat.h: A->preallocated = PETSC_FALSE; /* this was set to true for the MatSetValues_Hash() to work */ impls/aij/seq/seqhashmat.h:static PetscErrorCode MatDestroy_Seq_Hash(Mat A) impls/aij/seq/seqhashmat.h:static PetscErrorCode MatZeroEntries_Seq_Hash(Mat A) impls/aij/seq/seqhashmat.h:static PetscErrorCode MatSetRandom_Seq_Hash(Mat A, PetscRandom r) impls/aij/seq/seqhashmat.h:static PetscErrorCode MatSetUp_Seq_Hash(Mat A) impls/baij/mpi/mpibaij.c:/* defines MatSetValues_MPI_Hash(), MatAssemblyBegin_MPI_Hash(), and MatAssemblyEnd_MPI_Hash() */ impls/baij/seq/baij.c:/* defines MatSetValues_Seq_Hash(), MatAssemblyEnd_Seq_Hash(), MatSetUp_Seq_Hash() */ impls/sbaij/mpi/mpisbaij.c:/* defines MatSetValues_MPI_Hash(), MatAssemblyBegin_MPI_Hash(), MatAssemblyEnd_MPI_Hash(), MatSetUp_MPI_Hash() */ impls/sbaij/seq/sbaij.c:/* defines MatSetValues_Seq_Hash(), MatAssemblyEnd_Seq_Hash(), MatSetUp_Seq_Hash() */ Thanks for the numbers, it is good to see the performance is so similar to that obtained when providing preallocation information. > > Maybe can you tell me which file implements this function? > > Runfeng > > > > Runfeng Jin > ?2023?7?3??? 22:05??? >> Thank you for all your help! >> >> Runfeng >> >> Matthew Knepley > ?2023?7?3??? 22:03??? >>> On Mon, Jul 3, 2023 at 9:56?AM Runfeng Jin > wrote: >>>> Hi, impressive performance! >>>> I use the newest version of petsc(release branch), and it almost deletes all assembly and stash time in large processors (assembly time 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming cost, it really incredible. >>>> The order code has a regular arrangement of the number of nonzero-elements across rows, so I can have a good rough preallocation. And from the data, dedicatedly arrange data and roughly acquiring the max number of non-zero elements in rows can have a better performance than the new version without preallocation. However, in reality, I will use the newer version without preallocation for:1)less effort in programming and also nearly the same good performance 2) good memory usage(I see no unneeded memory after assembly) 3) dedicated preallocation is usually not very easy and cause extra time cost. >>>> Maybe it will be better that leave some space for the user to do a slight direction for the preallocation and thus acquire better performance. But have no idea how to direct it. >>>> And I am very curious about how petsc achieves this. How can it not know anything but achieve so good performance, and no wasted memory? May you have an explanation about this? >>> >>> We use a hash table to store the nonzeros on the fly, and then convert to packed storage on assembly. >>> >>> Thanks, >>> >>> Matt >>> >>>> assemble time: >>>> version\processors 4 8 16 32 64 128 256 >>>> old 14677s 4694s 1124s 572s 38s 8s 2s >>>> new 50s 28s 15s 7.8s 4s 2s 0.4s >>>> older 27s 24s 19s 12s 14s - - >>>> stash time(max among all processors): >>>> version\processors 4 8 16 32 64 128 256 >>>> old 3145s 2554s 673s 329s 201s 142s 138s >>>> new 2s 1s ~0s ~0s ~0s ~0s ~0s >>>> older 10s 73s 18s 5s 1s - - >>>> old: my poor preallocation >>>> new: newest version of petsc that do not preallocation >>>> older: the best preallocation version of my code. >>>> >>>> >>>> Runfeng >>>> >>>> Barry Smith > ?2023?7?3??? 12:19??? >>>>> >>>>> The main branch of PETSc now supports filling sparse matrices without providing any preallocation information. >>>>> >>>>> You can give it a try. Use your current fastest code but just remove ALL the preallocation calls. I would be interested in what kind of performance you get compared to your best current performance. >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>>> On Jul 2, 2023, at 11:24 PM, Runfeng Jin > wrote: >>>>>> >>>>>> Hi! Good advice! >>>>>> I set value with MatSetValues() API, which sets one part of a row at a time(I use a kind of tiling technology so I cannot get all values of a row at a time). >>>>>> I tested the number of malloc in these three cases. The number of mallocs is decreasing with the increase of processors, and all these are very large(the matrix is 283234*283234, as can see in the following). This should be due to the unqualified preallocation. I use a rough preallocation, that every processor counts the number of nonzero elements for the first 10 rows, and uses the largest one to preallocate memory for all local rows. It seems that not work well. >>>>>> number_of_processors number_of_max_mallocs_among_all_processors >>>>>> 64 20000 >>>>>> 128 17000 >>>>>> 256 11000 >>>>>> I change my way to preallocate. I evenly take 100 rows in every local matrix and take the largest one to preallocate memory for all local rows. Now the assemble time is reduced to a very small time. >>>>>> number_of_processors number_of_max_mallocs_among_all_processors >>>>>> 64 3000 >>>>>> 128 700 >>>>>> 256 500 >>>>>> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >>>>>> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>> 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 3 0 >>>>>> 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 3 0 >>>>>> 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 3 0 >>>>>> So the reason "why assemble time is smaller with the increasing number of processors " may be because more processors divide the malloc job so that total time is reduced? >>>>>> If so, I still have some questions: >>>>>> 1. If preallocation is not accurate, will the performance of the assembly be affected? I mean, when processors receive the elements that should be stored in their local by MPI, then will the new mallocs happen at this time point? >>>>>> 2. I can not give an accurate preallocation for the large cost, so is there any better way to preallocate for my situation? >>>>>> >>>>>> >>>>>> Barry Smith > ?2023?7?2??? 00:16??? >>>>>>> >>>>>>> I see no reason not to trust the times below, they seem reasonable. You get more than 2 times speed from 64 to 128 and then about 1.38 from 128 to 256. >>>>>>> >>>>>>> The total amount of data moved (number of messages moved times average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 to 2.3000e+09. A pretty moderate amount of data increase, but note that each time you double the number of ranks, you also increase substantially the network's hardware to move data, so one would hope for a good speed up. >>>>>>> >>>>>>> Also, the load balance is very good, near 1. Often with assembly, we see very out-of-balance, and it is difficult to get good speedup when the balance is really off. >>>>>>> >>>>>>> It looks like over 90% of the entire run time is coming from setting and assembling the values? Also the setting values time dominates assembly time more with more ranks. Are you setting a single value at a time or a collection of them? How big are the vectors? >>>>>>> >>>>>>> Run all three cases with -info :vec to see some information about how many mallocs where move to hold the stashed vector entries. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin > wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> Thanks for your reply. I try to use PetscLogEvent(), and the result shows same conclusion. >>>>>>>> What I have done is : >>>>>>>> ---------------- >>>>>>>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, Mat_setAsse_event; >>>>>>>> PetscClassId classid; >>>>>>>> PetscLogDouble user_event_flops; >>>>>>>> PetscClassIdRegister("Test assemble and set value", &classid); >>>>>>>> PetscLogEventRegister("Test only assemble", classid, &Mat_assemble_event); >>>>>>>> PetscLogEventRegister("Test only set values", classid, &Mat_setvalue_event); >>>>>>>> PetscLogEventRegister("Test both assemble and set values", classid, &Mat_setAsse_event); >>>>>>>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>>>>>>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>>>>>>> ...compute elements and use MatSetValues. No call for assembly >>>>>>>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>>>>>>> >>>>>>>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>>>>>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>>>>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>>>>>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>>>>>>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>>>>>>> ---------------- >>>>>>>> >>>>>>>> And the output as follows. By the way, dose petsc recorde all time between PetscLogEventBegin and PetscLogEventEnd? or just test the time of petsc API? >>>>>>> >>>>>>> It is all of the time. >>>>>>> >>>>>>>> ---------------- >>>>>>>> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >>>>>>>> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>>>> 64new 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>>>>>> 128new 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>>>>>> 256new 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>>>>>> >>>>>>>> 64: >>>>>>>> only assemble 1 1.0 2.6596e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>>>>>>> only setvalues 1 1.0 1.9987e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>>>>>>> Test both 1 1.0 4.6580e+02 1.0 0.00e+00 0.0 7.0e+03 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>>>>>>> >>>>>>>> 128: >>>>>>>> only assemble 1 1.0 6.9718e+01 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>>>>>>> only setvalues 1 1.0 1.4438e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>>>>>>> Test both 1 1.0 2.1417e+02 1.0 0.00e+00 0.0 2.6e+04 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>>>>>>> >>>>>>>> 256: >>>>>>>> only assemble 1 1.0 1.7482e+01 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>>>>>>> only setvalues 1 1.0 1.3717e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>>>>>>> Test both 1 1.0 1.5475e+02 1.0 0.00e+00 0.0 1.0e+05 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Runfeng >>>>>>>> >>>>>>>> Barry Smith > ?2023?6?30??? 23:35??? >>>>>>>>> >>>>>>>>> You cannot look just at the VecAssemblyEnd() time, that will very likely give the wrong impression of the total time it takes to put the values in. >>>>>>>>> >>>>>>>>> You need to register a new Event and put a PetscLogEvent() just before you start generating the vector entries and calling VecSetValues() and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the only way to get an accurate accounting of the time. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin > wrote: >>>>>>>>> > >>>>>>>>> > Hello! >>>>>>>>> > >>>>>>>>> > When I use PETSc build a sbaij matrix, I find a strange thing. When I increase the number of processors, the assemble time become smaller. All these are totally same matrix. The assemble time mainly arouse from message passing, which because I use dynamic workload that it is random for which elements are computed by which processor. >>>>>>>>> > But from instinct, if use more processors, then more possible that the processor computes elements storing in other processors. But from the output of log_view, It seems when use more processors, the processors compute more elements storing in its local(infer from that, with more processors, less total amount of passed messages). >>>>>>>>> > >>>>>>>>> > What could cause this happened? Thank you! >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Following is the output of log_view for 64\128\256 processors. Every row is time profiler of VecAssemblyEnd. >>>>>>>>> > >>>>>>>>> > ------------------------------------------------------------------------------------------------------------------------ >>>>>>>>> > processors Count Time (sec) Flop --- Global --- --- Stage ---- Total >>>>>>>>> > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>>>>> > 64 1 1.0 2.3775e+02 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>>>>>>> > 128 1 1.0 6.9945e+01 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>>>>>>> > 256 1 1.0 1.7445e+01 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>>>>>>> > >>>>>>>>> > Runfeng Jin >>>>>>>>> >>>>>>> >>>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Mon Jul 3 10:03:23 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Mon, 3 Jul 2023 23:03:23 +0800 Subject: [petsc-users] Smaller assemble time with increasing processors In-Reply-To: <1D1523EB-62D4-451D-9961-9E6FAA05F1D2@petsc.dev> References: <4E68029B-EC0E-4C92-B474-4B997DDE972C@petsc.dev> <0AC53DB6-5228-4FD6-8369-9467FCC9758F@petsc.dev> <1D1523EB-62D4-451D-9961-9E6FAA05F1D2@petsc.dev> Message-ID: Thank you very much?! Runfeng Barry Smith ?2023?7?3??? 22:52??? > > > On Jul 3, 2023, at 10:11 AM, Runfeng Jin wrote: > > Hi, > >> We use a hash table to store the nonzeros on the fly, and then convert to >> packed storage on assembly. >> > > There is "extra memory" since the matrix entries are first stored in a > hash and then converted into the regular CSR format, so for a short while, > both copies are in memory. > > We use the amazing khash package, include/petsc/private/khash/khash.h, > our code is scattered around a bit depending on the matrix format we will > be forming. > > cd src/mat > > git grep "_Hash(" > > impls/aij/mpi/mpiaij.c:/* defines MatSetValues_MPI*_Hash(*), > MatAssemblyBegin_MPI*_Hash(*), and MatAssemblyEnd_MPI*_Hash(*) */ > > impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:/* defines > MatSetValues_MPICUSPARSE*_Hash(*) */ > > impls/aij/mpi/mpicusparse/mpiaijcusparse.cu: PetscCall(MatSetUp_MPI > *_Hash(*A)); > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetValues_MPI*_Hash(*Mat > A, PetscInt m, const PetscInt *rows, PetscInt n, const PetscInt *cols, > const PetscScalar *values, InsertMode addv) > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatAssemblyBegin_MPI > *_Hash(*Mat A, PETSC_UNUSED MatAssemblyType type) > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatAssemblyEnd_MPI > *_Hash(*Mat A, MatAssemblyType type) > > impls/aij/mpi/mpihashmat.h: PetscCall(MatSetValues_MPI*_Hash(*A, > 1, row + i, ncols, col + i, val + i, A->insertmode)); > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatDestroy_MPI*_Hash(*Mat > A) > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatZeroEntries_MPI > *_Hash(*PETSC_UNUSED Mat A) > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetRandom_MPI*_Hash(*Mat > A, PETSC_UNUSED PetscRandom r) > > impls/aij/mpi/mpihashmat.h:static PetscErrorCode MatSetUp_MPI*_Hash(*Mat > A) > > impls/aij/seq/aij.c:/* defines MatSetValues_Seq*_Hash(*), > MatAssemblyEnd_Seq*_Hash(*), MatSetUp_Seq*_Hash(*) */ > > impls/aij/seq/seqhashmat.h:static PetscErrorCode MatAssemblyEnd_Seq > *_Hash(*Mat A, MatAssemblyType type) > > impls/aij/seq/seqhashmat.h: A->preallocated = PETSC_FALSE; /* this was > set to true for the MatSetValues*_Hash(*) to work */ > > impls/aij/seq/seqhashmat.h:static PetscErrorCode MatDestroy_Seq*_Hash(*Mat > A) > > impls/aij/seq/seqhashmat.h:static PetscErrorCode MatZeroEntries_Seq > *_Hash(*Mat A) > > impls/aij/seq/seqhashmat.h:static PetscErrorCode MatSetRandom_Seq*_Hash(*Mat > A, PetscRandom r) > > impls/aij/seq/seqhashmat.h:static PetscErrorCode MatSetUp_Seq*_Hash(*Mat > A) > > impls/baij/mpi/mpibaij.c:/* defines MatSetValues_MPI*_Hash(*), > MatAssemblyBegin_MPI*_Hash(*), and MatAssemblyEnd_MPI*_Hash(*) */ > > impls/baij/seq/baij.c:/* defines MatSetValues_Seq*_Hash(*), > MatAssemblyEnd_Seq*_Hash(*), MatSetUp_Seq*_Hash(*) */ > > impls/sbaij/mpi/mpisbaij.c:/* defines MatSetValues_MPI*_Hash(*), > MatAssemblyBegin_MPI*_Hash(*), MatAssemblyEnd_MPI*_Hash(*), MatSetUp_MPI > *_Hash(*) */ > > impls/sbaij/seq/sbaij.c:/* defines MatSetValues_Seq*_Hash(*), > MatAssemblyEnd_Seq*_Hash(*), MatSetUp_Seq*_Hash(*) */ > > Thanks for the numbers, it is good to see the performance is so similar to > that obtained when providing preallocation information. > > > Maybe can you tell me which file implements this function? > > Runfeng > > > > Runfeng Jin ?2023?7?3??? 22:05??? > >> Thank you for all your help! >> >> Runfeng >> >> Matthew Knepley ?2023?7?3??? 22:03??? >> >>> On Mon, Jul 3, 2023 at 9:56?AM Runfeng Jin wrote: >>> >>>> Hi, impressive performance! >>>> I use the newest version of petsc(release branch), and it almost >>>> deletes all assembly and stash time in large processors (assembly time >>>> 64-4s/128-2s/256-0.2s, stash time all below 2s). For the zero programming >>>> cost, it really incredible. >>>> The order code has a regular arrangement of the number of >>>> nonzero-elements across rows, so I can have a good rough preallocation. And >>>> from the data, dedicatedly arrange data and roughly acquiring the max >>>> number of non-zero elements in rows can have a better performance than the >>>> new version without preallocation. However, in reality, I will use the >>>> newer version without preallocation for:1)less effort in programming and >>>> also nearly the same good performance 2) good memory usage(I see no >>>> unneeded memory after assembly) 3) dedicated preallocation is usually not >>>> very easy and cause extra time cost. >>>> Maybe it will be better that leave some space for the user to do a >>>> slight direction for the preallocation and thus acquire better performance. >>>> But have no idea how to direct it. >>>> And I am very curious about how petsc achieves this. How can it not >>>> know anything but achieve so good performance, and no wasted memory? May >>>> you have an explanation about this? >>>> >>> >>> We use a hash table to store the nonzeros on the fly, and then convert >>> to packed storage on assembly. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> assemble time: >>>> version\processors 4 8 16 32 >>>> 64 128 256 >>>> old 14677s 4694s 1124s 572s >>>> 38s 8s 2s >>>> new 50s 28s 15s >>>> 7.8s 4s 2s 0.4s >>>> older 27s 24s 19s >>>> 12s 14s - - >>>> stash time(max among all processors): >>>> version\processors 4 8 16 32 >>>> 64 128 256 >>>> old 3145s 2554s 673s 329s >>>> 201s 142s 138s >>>> new 2s 1s ~0s >>>> ~0s ~0s ~0s ~0s >>>> older 10s 73s 18s >>>> 5s 1s - - >>>> old: my poor preallocation >>>> new: newest version of petsc that do not preallocation >>>> older: the best preallocation version of my code. >>>> >>>> >>>> Runfeng >>>> >>>> Barry Smith ?2023?7?3??? 12:19??? >>>> >>>>> >>>>> The main branch of PETSc now supports filling sparse matrices >>>>> without providing any preallocation information. >>>>> >>>>> You can give it a try. Use your current fastest code but just >>>>> remove ALL the preallocation calls. I would be interested in what kind of >>>>> performance you get compared to your best current performance. >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> On Jul 2, 2023, at 11:24 PM, Runfeng Jin wrote: >>>>> >>>>> Hi! Good advice! >>>>> I set value with MatSetValues() API, which sets one part of a row >>>>> at a time(I use a kind of tiling technology so I cannot get all values of a >>>>> row at a time). >>>>> I tested the number of malloc in these three cases. The number of >>>>> mallocs is decreasing with the increase of processors, and all these are >>>>> very large(the matrix is 283234*283234, as can see in the following). This >>>>> should be due to the unqualified preallocation. I use a rough >>>>> preallocation, that every processor counts the number of nonzero elements >>>>> for the first 10 rows, and uses the largest one to preallocate memory for >>>>> all local rows. It seems that not work well. >>>>> >>>>> number_of_processors number_of_max_mallocs_among_all_processors >>>>> 64 20000 >>>>> 128 17000 >>>>> 256 11000 >>>>> >>>>> I change my way to preallocate. I evenly take 100 rows in every >>>>> local matrix and take the largest one to preallocate memory for all local >>>>> rows. Now the assemble time is reduced to a very small time. >>>>> >>>>> number_of_processors number_of_max_mallocs_among_all_processors >>>>> 64 3000 >>>>> 128 700 >>>>> 256 500 >>>>> >>>>> Event Count Time (sec) Flop >>>>> --- Global --- >>>>> --- Stage ---- Total >>>>> Max Ratio Max Ratio Max Ratio >>>>> Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R >>>>> Mflop/s >>>>> 64 1 1.0 3.8999e+01 1.0 0.00e+00 0.0 >>>>> 7.1e+03 2.9e+05 1.1e+01 15 0 1 8 3 15 0 1 8 >>>>> 3 0 >>>>> >>>>> 128 1 1.0 8.5714e+00 1.0 0.00e+00 0.0 >>>>> 2.6e+04 8.1e+04 1.1e+01 5 0 1 4 3 5 0 1 4 >>>>> 3 0 >>>>> 256 1 1.0 2.5512e+00 1.0 0.00e+00 0.0 >>>>> 1.0e+05 2.3e+04 1.1e+01 2 0 1 3 3 2 0 1 3 >>>>> 3 0 >>>>> >>>>> So the reason "why assemble time is smaller with the increasing number >>>>> of processors " may be because more processors divide the malloc job so >>>>> that total time is reduced? >>>>> If so, I still have some questions: >>>>> 1. If preallocation is not accurate, will the performance of the >>>>> assembly be affected? I mean, when processors receive the elements that >>>>> should be stored in their local by MPI, then will the new mallocs happen >>>>> at this time point? >>>>> 2. I can not give an accurate preallocation for the large cost, so >>>>> is there any better way to preallocate for my situation? >>>>> >>>>> >>>>> >>>>> Barry Smith ?2023?7?2??? 00:16??? >>>>> >>>>>> >>>>>> I see no reason not to trust the times below, they seem >>>>>> reasonable. You get more than 2 times speed from 64 to 128 and then about >>>>>> 1.38 from 128 to 256. >>>>>> >>>>>> The total amount of data moved (number of messages moved times >>>>>> average length) goes from 7.0e+03 * 2.8e+05 1.9600e+09 to 2.1060e+09 >>>>>> to 2.3000e+09. A pretty moderate amount of data increase, but note that >>>>>> each time you double the number of ranks, you also increase substantially >>>>>> the network's hardware to move data, so one would hope for a good speed up. >>>>>> >>>>>> Also, the load balance is very good, near 1. Often with assembly, >>>>>> we see very out-of-balance, and it is difficult to get good speedup when >>>>>> the balance is really off. >>>>>> >>>>>> It looks like over 90% of the entire run time is coming from >>>>>> setting and assembling the values? Also the setting values time dominates >>>>>> assembly time more with more ranks. Are you setting a single value at a >>>>>> time or a collection of them? How big are the vectors? >>>>>> >>>>>> Run all three cases with -info :vec to see some information about >>>>>> how many mallocs where move to hold the stashed vector entries. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Jun 30, 2023, at 10:25 PM, Runfeng Jin >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> Thanks for your reply. I try to use PetscLogEvent(), and the >>>>>> result shows same conclusion. >>>>>> What I have done is : >>>>>> ---------------- >>>>>> PetscLogEvent Mat_assemble_event, Mat_setvalue_event, >>>>>> Mat_setAsse_event; >>>>>> PetscClassId classid; >>>>>> PetscLogDouble user_event_flops; >>>>>> PetscClassIdRegister("Test assemble and set value", &classid); >>>>>> PetscLogEventRegister("Test only assemble", classid, >>>>>> &Mat_assemble_event); >>>>>> PetscLogEventRegister("Test only set values", classid, >>>>>> &Mat_setvalue_event); >>>>>> PetscLogEventRegister("Test both assemble and set values", >>>>>> classid, &Mat_setAsse_event); >>>>>> PetscLogEventBegin(Mat_setAsse_event, 0, 0, 0, 0); >>>>>> PetscLogEventBegin(Mat_setvalue_event, 0, 0, 0, 0); >>>>>> ...compute elements and use MatSetValues. No call for assembly >>>>>> PetscLogEventEnd(Mat_setvalue_event, 0, 0, 0, 0); >>>>>> >>>>>> PetscLogEventBegin(Mat_assemble_event, 0, 0, 0, 0); >>>>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>>>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>>>>> PetscLogEventEnd(Mat_assemble_event, 0, 0, 0, 0); >>>>>> PetscLogEventEnd(Mat_setAsse_event, 0, 0, 0, 0); >>>>>> ---------------- >>>>>> >>>>>> And the output as follows. By the way, dose petsc recorde all >>>>>> time between PetscLogEventBegin and PetscLogEventEnd? or just test the time >>>>>> of petsc API? >>>>>> >>>>>> >>>>>> It is all of the time. >>>>>> >>>>>> ---------------- >>>>>> Event Count Time (sec) Flop >>>>>> --- Global --- --- Stage ---- Total >>>>>> Max Ratio *Max* Ratio Max Ratio Mess >>>>>> AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>> 64new 1 1.0 *2.3775e+02* 1.0 0.00e+00 0.0 6.2e+03 >>>>>> 2.3e+04 9.0e+00 52 0 1 1 2 52 0 1 1 2 0 >>>>>> 128new 1 1.0* 6.9945e+01* 1.0 0.00e+00 0.0 2.5e+04 >>>>>> 1.1e+04 9.0e+00 30 0 1 1 2 30 0 1 1 2 0 >>>>>> 256new 1 1.0 *1.7445e+01* 1.0 0.00e+00 0.0 9.9e+04 >>>>>> 5.2e+03 9.0e+00 10 0 1 1 2 10 0 1 1 2 0 >>>>>> >>>>>> 64: >>>>>> only assemble 1 1.0 *2.6596e+02 *1.0 0.00e+00 0.0 7.0e+03 >>>>>> 2.8e+05 1.1e+01 55 0 1 8 3 55 0 1 8 3 0 >>>>>> only setvalues 1 1.0 *1.9987e+02* 1.0 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >>>>>> Test both 1 1.0 4.*6580e+02* 1.0 0.00e+00 0.0 7.0e+03 >>>>>> 2.8e+05 1.5e+01 96 0 1 8 4 96 0 1 8 4 0 >>>>>> >>>>>> 128: >>>>>> only assemble 1 1.0 *6.9718e+01* 1.0 0.00e+00 0.0 2.6e+04 >>>>>> 8.1e+04 1.1e+01 30 0 1 4 3 30 0 1 4 3 0 >>>>>> only setvalues 1 1.0 *1.4438e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 60 0 0 0 0 60 0 0 0 0 0 >>>>>> Test both 1 1.0 *2.1417e+02* 1.0 0.00e+00 0.0 2.6e+04 >>>>>> 8.1e+04 1.5e+01 91 0 1 4 4 91 0 1 4 4 0 >>>>>> >>>>>> 256: >>>>>> only assemble 1 1.0 *1.7482e+01* 1.0 0.00e+00 0.0 1.0e+05 >>>>>> 2.3e+04 1.1e+01 10 0 1 3 3 10 0 1 3 3 0 >>>>>> only setvalues 1 1.0 *1.3717e+02* 1.1 0.00e+00 0.0 0.0e+00 >>>>>> 0.0e+00 0.0e+00 78 0 0 0 0 78 0 0 0 0 0 >>>>>> Test both 1 1.0 *1.5475e+02* 1.0 0.00e+00 0.0 1.0e+05 >>>>>> 2.3e+04 1.5e+01 91 0 1 3 4 91 0 1 3 4 0 >>>>>> >>>>>> >>>>>> >>>>>> Runfeng >>>>>> >>>>>> Barry Smith ?2023?6?30??? 23:35??? >>>>>> >>>>>>> >>>>>>> You cannot look just at the VecAssemblyEnd() time, that will very >>>>>>> likely give the wrong impression of the total time it takes to put the >>>>>>> values in. >>>>>>> >>>>>>> You need to register a new Event and put a PetscLogEvent() just >>>>>>> before you start generating the vector entries and calling VecSetValues() >>>>>>> and put the PetscLogEventEnd() just after the VecAssemblyEnd() this is the >>>>>>> only way to get an accurate accounting of the time. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> > On Jun 30, 2023, at 11:21 AM, Runfeng Jin >>>>>>> wrote: >>>>>>> > >>>>>>> > Hello! >>>>>>> > >>>>>>> > When I use PETSc build a sbaij matrix, I find a strange thing. >>>>>>> When I increase the number of processors, the assemble time become smaller. >>>>>>> All these are totally same matrix. The assemble time mainly arouse from >>>>>>> message passing, which because I use dynamic workload that it is random for >>>>>>> which elements are computed by which processor. >>>>>>> > But from instinct, if use more processors, then more possible that >>>>>>> the processor computes elements storing in other processors. But from the >>>>>>> output of log_view, It seems when use more processors, the processors >>>>>>> compute more elements storing in its local(infer from that, with more >>>>>>> processors, less total amount of passed messages). >>>>>>> > >>>>>>> > What could cause this happened? Thank you! >>>>>>> > >>>>>>> > >>>>>>> > Following is the output of log_view for 64\128\256 processors. >>>>>>> Every row is time profiler of VecAssemblyEnd. >>>>>>> > >>>>>>> > >>>>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>>>> > processors Count Time (sec) >>>>>>> Flop >>>>>>> --- Global --- --- >>>>>>> Stage ---- Total >>>>>>> > Max Ratio Max >>>>>>> Ratio Max Ratio Mess AvgLen >>>>>>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>>> > 64 1 1.0 2.3775e+02 >>>>>>> 1.0 0.00e+00 0.0 6.2e+03 2.3e+04 9.0e+00 >>>>>>> 52 0 1 1 2 52 0 1 1 2 >>>>>>> 0 >>>>>>> > 128 1 1.0 6.9945e+01 >>>>>>> 1.0 0.00e+00 0.0 2.5e+04 1.1e+04 9.0e+00 >>>>>>> 30 0 1 1 2 30 0 1 1 2 >>>>>>> 0 >>>>>>> > 256 1 1.0 1.7445e+01 >>>>>>> 1.0 0.00e+00 0.0 9.9e+04 5.2e+03 9.0e+00 >>>>>>> 10 0 1 1 2 10 0 1 1 2 >>>>>>> 0 >>>>>>> > >>>>>>> > Runfeng Jin >>>>>>> >>>>>>> >>>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boga.keo at gmail.com Tue Jul 4 13:24:08 2023 From: boga.keo at gmail.com (Ju Taur) Date: Wed, 5 Jul 2023 00:24:08 +0600 Subject: [petsc-users] Basic Inquiries about PETSc Message-ID: Hi, Everyone. I am a new PETSc user and have little knowledge about it. So my queries are as follows: 1. Is PETSc only a library for solving PDEs, or is itself a software that has a graphical user interface like MATLAB or other software? 2. Is there any video tutorial for installing it step by step, as it seemed somewhat difficult to install by reading the instructions available on its website? 3. Is there any video link showing the simulation and solving a PDE using PETSc? I am cordially sorry for asking these silly questions, but I need to know these. Please forgive me, and I would be grateful if you could answer my questions. Thank you, Taur Kirmin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 4 14:16:29 2023 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 4 Jul 2023 15:16:29 -0400 Subject: [petsc-users] Basic Inquiries about PETSc In-Reply-To: References: Message-ID: In googling 'petsc tutorials youtube' you get several videos. The first one, for me, is a good place to start: https://www.youtube.com/watch?v=u7IjOrrS_ro&list=PLz9uieREhSLNqjHj-_YD2F9-n1upvIsw1&index=1 Enjoy, Mark On Tue, Jul 4, 2023 at 2:29?PM Ju Taur wrote: > Hi, > Everyone. I am a new PETSc user and have little knowledge about it. So my > queries are as follows: > 1. Is PETSc only a library for solving PDEs, or is itself a software that > has a graphical user interface like MATLAB or other software? > 2. Is there any video tutorial for installing it step by step, as it > seemed somewhat difficult to install by reading the instructions available > on its website? > 3. Is there any video link showing the simulation and solving a PDE using > PETSc? > > I am cordially sorry for asking these silly questions, but I need to know > these. Please forgive me, and I would be grateful if you could answer my > questions. > > Thank you, > Taur Kirmin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 4 14:40:53 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 4 Jul 2023 15:40:53 -0400 Subject: [petsc-users] Basic Inquiries about PETSc In-Reply-To: References: Message-ID: You might consider https://www.firedrakeproject.org/ it wraps PETSc in a higher-level environment for solving PDEs. > On Jul 4, 2023, at 2:24 PM, Ju Taur wrote: > > Hi, > Everyone. I am a new PETSc user and have little knowledge about it. So my queries are as follows: > 1. Is PETSc only a library for solving PDEs, or is itself a software that has a graphical user interface like MATLAB or other software? > 2. Is there any video tutorial for installing it step by step, as it seemed somewhat difficult to install by reading the instructions available on its website? > 3. Is there any video link showing the simulation and solving a PDE using PETSc? > > I am cordially sorry for asking these silly questions, but I need to know these. Please forgive me, and I would be grateful if you could answer my questions. > > Thank you, > Taur Kirmin -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.khurana22 at imperial.ac.uk Wed Jul 5 10:28:50 2023 From: p.khurana22 at imperial.ac.uk (Khurana, Parv) Date: Wed, 5 Jul 2023 15:28:50 +0000 Subject: [petsc-users] Running AMG libraries as standalone solvers Message-ID: Hello PETSc users, I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples here https://petsc.org/main/tutorials/handson/#, but it hasn't helped me. Any hints on how to use AMG directly as a solver? Best Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 5 10:45:45 2023 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 5 Jul 2023 11:45:45 -0400 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: Message-ID: On Wed, Jul 5, 2023 at 11:29?AM Khurana, Parv wrote: > Hello PETSc users, > > > > I am fairly inexperienced with PETSc as of now and am new to the mailing > list! Thanks for running this channel. > > > > I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am > trying to compare the performance of solving a Poisson problem using a AMG > Preconditioned GMRES iterative solve vs using AMG as the solver. I use > PETSc options using the options database keys as of now, and it is > connected to a flow solver (Nektar++) I use for my research. > > > > I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type > gmres and then specifying the preconditioner I want using, for e.g, -pc_type > hypre -pc_hypre_type boomeramg. > Looks good. You need to configure with --download-hypre and you should be good to go. You can also try -pc_type gamg You can also try -pc_type ml If I want to use the AMG routine, I am currently setting -ksp_type preonly > and the same -pc_type. > I am not understanding this. You just said you use -ksp_type gmres. Stick with that! > However, I am not sure if this is correct way to go about it due to two > reasons: a) my solution using AMG as a solver with these options has a > larger error than AMG Preconditioned GMRES (which could still be > acceptable), and b) I could not find any clear documentation regarding how > to use AMG directly as a solver. I saw some hints in the examples here > https://petsc.org/main/tutorials/handson/# > , but it hasn?t helped me. > > > There are a lot of solver examples in the code base. eg, src/snes/tutorials/ex19.c: args: -da_refine 5 -snes_monitor -snes_converged_reason -pc_type hypre -dm_mat_type {{aij baij}} There are regression tests in a big comment block at the end of each example file. You can use these as examples. Good luck, Mark > Any hints on how to use AMG directly as a solver? > > > > Best > > Parv > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 10:46:36 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 11:46:36 -0400 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: Message-ID: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> -ksp_type richardson If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer > On Jul 5, 2023, at 11:28 AM, Khurana, Parv wrote: > > Hello PETSc users, > > I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. > > I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. > > I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/# , but it hasn?t helped me. > > Any hints on how to use AMG directly as a solver? > > Best > Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 10:51:22 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 11:51:22 -0400 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> References: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> Message-ID: BTW: in my experience Geometric multigrid generally works great as a standalone solver, that is accelerating it with a Krylov method is not needed, and while it may improve the convergence rate slightly, it may end up taking a bit more time then not using the Krylov solver (due to the extra Krylov solver overhead). Algebraic multigrid is usually run with a Krylov solver and running without a Krylov solver, as far as I am aware, generally performs poorly. > On Jul 5, 2023, at 11:46 AM, Barry Smith wrote: > > > -ksp_type richardson > > If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer > > > >> On Jul 5, 2023, at 11:28 AM, Khurana, Parv wrote: >> >> Hello PETSc users, >> >> I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. >> >> I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. >> >> I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/# , but it hasn?t helped me. >> >> Any hints on how to use AMG directly as a solver? >> >> Best >> Parv > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.khurana22 at imperial.ac.uk Wed Jul 5 11:09:28 2023 From: p.khurana22 at imperial.ac.uk (Khurana, Parv) Date: Wed, 5 Jul 2023 16:09:28 +0000 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> Message-ID: Thank you Mark, Barry for the advice. -ksp_type richardson may be what I am looking for. I wanted a simple iterative scheme that allows AMG to be the main work horse ? this allows that. On using -ksp_type preonly, I observed only 1 KSP iteration and it does not drop the residual norm to my desired level (-ksp_rtol 1e-7). Whereas when I use gmres or Richardson I see 6-10 ksp iterations and the desired residual drop. Is it possible to increase (and fix) the number of KSP iterations with preonly? You mentioned below that it gives only one cycle so I presume it is not possible. Geometric mg is not a feasible option for my application as I will be ultimately dealing with 3D, unstructured meshes for hp-finite element methods. Some clever tweaks may allow it (like having sub-structured meshes) but not as of now. Best Parv From: Barry Smith Sent: 05 July 2023 16:51 To: Khurana, Parv Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Running AMG libraries as standalone solvers This email from bsmith at petsc.dev originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address. BTW: in my experience Geometric multigrid generally works great as a standalone solver, that is accelerating it with a Krylov method is not needed, and while it may improve the convergence rate slightly, it may end up taking a bit more time then not using the Krylov solver (due to the extra Krylov solver overhead). Algebraic multigrid is usually run with a Krylov solver and running without a Krylov solver, as far as I am aware, generally performs poorly. On Jul 5, 2023, at 11:46 AM, Barry Smith > wrote: -ksp_type richardson If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer On Jul 5, 2023, at 11:28 AM, Khurana, Parv > wrote: Hello PETSc users, I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/#, but it hasn?t helped me. Any hints on how to use AMG directly as a solver? Best Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.khurana22 at imperial.ac.uk Wed Jul 5 12:18:57 2023 From: p.khurana22 at imperial.ac.uk (Khurana, Parv) Date: Wed, 5 Jul 2023 17:18:57 +0000 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> Message-ID: Hi Barry, Using -ksp_type preonly,and tweaking -pc_hypre_boomeramg_max_iter and -pc_hypre_boomeramg_tol according to my needs seems to have done the trick for me. Credit: https://petsc.org/release/faq/#i-want-to-use-hypre-boomeramg-without-gmres-but-when-i-run-pc-type-hypre-pc-hypre-type-boomeramg-ksp-type-preonly-i-don-t-get-a-very-accurate-answer Best Parv From: petsc-users On Behalf Of Khurana, Parv Sent: 05 July 2023 17:09 To: Barry Smith Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Running AMG libraries as standalone solvers Thank you Mark, Barry for the advice. -ksp_type richardson may be what I am looking for. I wanted a simple iterative scheme that allows AMG to be the main work horse ? this allows that. On using -ksp_type preonly, I observed only 1 KSP iteration and it does not drop the residual norm to my desired level (-ksp_rtol 1e-7). Whereas when I use gmres or Richardson I see 6-10 ksp iterations and the desired residual drop. Is it possible to increase (and fix) the number of KSP iterations with preonly? You mentioned below that it gives only one cycle so I presume it is not possible. Geometric mg is not a feasible option for my application as I will be ultimately dealing with 3D, unstructured meshes for hp-finite element methods. Some clever tweaks may allow it (like having sub-structured meshes) but not as of now. Best Parv From: Barry Smith > Sent: 05 July 2023 16:51 To: Khurana, Parv > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Running AMG libraries as standalone solvers This email from bsmith at petsc.dev originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address. BTW: in my experience Geometric multigrid generally works great as a standalone solver, that is accelerating it with a Krylov method is not needed, and while it may improve the convergence rate slightly, it may end up taking a bit more time then not using the Krylov solver (due to the extra Krylov solver overhead). Algebraic multigrid is usually run with a Krylov solver and running without a Krylov solver, as far as I am aware, generally performs poorly. On Jul 5, 2023, at 11:46 AM, Barry Smith > wrote: -ksp_type richardson If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer On Jul 5, 2023, at 11:28 AM, Khurana, Parv > wrote: Hello PETSc users, I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/#, but it hasn?t helped me. Any hints on how to use AMG directly as a solver? Best Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 12:56:56 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 13:56:56 -0400 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> Message-ID: <094D219E-646A-48F1-9309-FC5187DE341B@petsc.dev> > On Jul 5, 2023, at 1:18 PM, Khurana, Parv wrote: > > Hi Barry, > > Using -ksp_type preonly,and tweaking -pc_hypre_boomeramg_max_iter and -pc_hypre_boomeramg_tol according to my needs seems to have done the trick for me. Sure, but there is no particular advantage (or disadvantage that I know of) in doing it this way. It is just a question of whether PETSc (via -ksp_type richardson) or hypre (via -pc_hypre_boomeramg_max_iter ) controls the simple iteration (what we call Richardson in PETSc) that wraps around the V-cycle. I would be surprised if the solve time between -ksp_typre richardson and -pc_hypre_boomeramg_max_iter and -pc_hypre_boomeramg_tol with similar tolerances was different significantly. > > Credit: https://petsc.org/release/faq/#i-want-to-use-hypre-boomeramg-without-gmres-but-when-i-run-pc-type-hypre-pc-hypre-type-boomeramg-ksp-type-preonly-i-don-t-get-a-very-accurate-answer > > Best > Parv > > From: petsc-users > On Behalf Of Khurana, Parv > Sent: 05 July 2023 17:09 > To: Barry Smith > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Running AMG libraries as standalone solvers > > Thank you Mark, Barry for the advice. > > -ksp_type richardson may be what I am looking for. I wanted a simple iterative scheme that allows AMG to be the main work horse ? this allows that. > > On using -ksp_type preonly, I observed only 1 KSP iteration and it does not drop the residual norm to my desired level (-ksp_rtol 1e-7). Whereas when I use gmres or Richardson I see 6-10 ksp iterations and the desired residual drop. Is it possible to increase (and fix) the number of KSP iterations with preonly? You mentioned below that it gives only one cycle so I presume it is not possible. > > Geometric mg is not a feasible option for my application as I will be ultimately dealing with 3D, unstructured meshes for hp-finite element methods. Some clever tweaks may allow it (like having sub-structured meshes) but not as of now. > > Best > Parv > > > > From: Barry Smith > > Sent: 05 July 2023 16:51 > To: Khurana, Parv > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Running AMG libraries as standalone solvers > > This email from bsmith at petsc.dev originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address. > > > > BTW: in my experience > > Geometric multigrid generally works great as a standalone solver, that is accelerating it with a Krylov method is not needed, and while it may improve the convergence rate slightly, it may end up taking a bit more time then not using the Krylov solver (due to the extra Krylov solver overhead). > > Algebraic multigrid is usually run with a Krylov solver and running without a Krylov solver, as far as I am aware, generally performs poorly. > > > > On Jul 5, 2023, at 11:46 AM, Barry Smith > wrote: > > > -ksp_type richardson > > If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer > > > > > On Jul 5, 2023, at 11:28 AM, Khurana, Parv > wrote: > > Hello PETSc users, > > I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. > > I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. > > I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/# , but it hasn?t helped me. > > Any hints on how to use AMG directly as a solver? > > Best > Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 12:59:42 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 13:59:42 -0400 Subject: [petsc-users] Running AMG libraries as standalone solvers In-Reply-To: References: <10712F33-ED78-494A-B77F-2027C1B74D4B@petsc.dev> Message-ID: <29D1484E-86FD-45A1-A484-A514D25BFDE5@petsc.dev> We call simple iteration in PETSc Richardson, while preonly is defined as "apply the preconditioner once" so -ksp_type richardson -pc_type gamg is what you want to try, there is no "tweaks" for -ksp_type preonly to turn it into simple iteration since -ksp_type richardson is exactly for using simple iteration. > On Jul 5, 2023, at 12:09 PM, Khurana, Parv wrote: > > Thank you Mark, Barry for the advice. > > -ksp_type richardson may be what I am looking for. I wanted a simple iterative scheme that allows AMG to be the main work horse ? this allows that. > On using -ksp_type preonly, I observed only 1 KSP iteration and it does not drop the residual norm to my desired level (-ksp_rtol 1e-7). Whereas when I use gmres or Richardson I see 6-10 ksp iterations and the desired residual drop. Is it possible to increase (and fix) the number of KSP iterations with preonly? You mentioned below that it gives only one cycle so I presume it is not possible. > > Geometric mg is not a feasible option for my application as I will be ultimately dealing with 3D, unstructured meshes for hp-finite element methods. Some clever tweaks may allow it (like having sub-structured meshes) but not as of now. > > Best > Parv > > > > From: Barry Smith > > Sent: 05 July 2023 16:51 > To: Khurana, Parv > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Running AMG libraries as standalone solvers > > This email from bsmith at petsc.dev originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list to disable email stamping for this address. > > > > BTW: in my experience > > Geometric multigrid generally works great as a standalone solver, that is accelerating it with a Krylov method is not needed, and while it may improve the convergence rate slightly, it may end up taking a bit more time then not using the Krylov solver (due to the extra Krylov solver overhead). > > Algebraic multigrid is usually run with a Krylov solver and running without a Krylov solver, as far as I am aware, generally performs poorly. > > > > On Jul 5, 2023, at 11:46 AM, Barry Smith > wrote: > > > -ksp_type richardson > > If you use -ksp_typre preonly you effectively get one V-cycle (or whatever cycle type you may have selected) which in general, will give you a more or less garbage answer > > > > > On Jul 5, 2023, at 11:28 AM, Khurana, Parv > wrote: > > Hello PETSc users, > > I am fairly inexperienced with PETSc as of now and am new to the mailing list! Thanks for running this channel. > > I seek basic help regarding running AMG routines (BoomerAMG/ML/GAMG). I am trying to compare the performance of solving a Poisson problem using a AMG Preconditioned GMRES iterative solve vs using AMG as the solver. I use PETSc options using the options database keys as of now, and it is connected to a flow solver (Nektar++) I use for my research. > > I currently run the AMG Preconditioned GMRES iterative solve by setting -ksp_type gmres and then specifying the preconditioner I want using, for e.g, -pc_type hypre -pc_hypre_type boomeramg. If I want to use the AMG routine, I am currently setting -ksp_type preonly and the same -pc_type. However, I am not sure if this is correct way to go about it due to two reasons: a) my solution using AMG as a solver with these options has a larger error than AMG Preconditioned GMRES (which could still be acceptable), and b) I could not find any clear documentation regarding how to use AMG directly as a solver. I saw some hints in the examples herehttps://petsc.org/main/tutorials/handson/# , but it hasn?t helped me. > > Any hints on how to use AMG directly as a solver? > > Best > Parv -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuhongrui at utexas.edu Wed Jul 5 19:54:55 2023 From: yuhongrui at utexas.edu (Hongrui Yu) Date: Wed, 5 Jul 2023 19:54:55 -0500 Subject: [petsc-users] Abort SNES Mid-iteration Message-ID: <065d01d9afa4$7b45f560$71d1e020$@utexas.edu> Hello PETSc users, I'd like to use the error handling mechanism to abort SNES mid-iteration if certain quantities become nonphysical by calling SETERRQ() inside FormFunction() or FormJacobian(). I'm wondering if users can define their own error handler routines to use with PetscPushErrorHandler() and PetscPopErrorHandler()? I'm also open to any suggestion on better ways to quit SNES mid-iteration. Thanks in advance for any help or advice! Hongrui -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 20:22:53 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 21:22:53 -0400 Subject: [petsc-users] Abort SNES Mid-iteration In-Reply-To: <065d01d9afa4$7b45f560$71d1e020$@utexas.edu> References: <065d01d9afa4$7b45f560$71d1e020$@utexas.edu> Message-ID: I assume you want to stop the SNES solve, but you do not want the program to end immediately? Instead, you want SNESSolve() to end immediately with a very useful message? This is done with https://petsc.org/release/manualpages/SNES/SNESSetFunctionDomainError/#snessetfunctiondomainerror or https://petsc.org/release/manualpages/SNES/SNESSetJacobianDomainError/#snessetjacobiandomainerror In conjunction with using this, you must call https://petsc.org/release/manualpages/SNES/SNESGetConvergedReason/#snesgetconvergedreason immediately after each SNESSolve() and check if the reason is negative. The specific values SNES_DIVERGED_FUNCTION_DOMAIN will be set if you had called SNESSetFunctionDomainError() and SNES_DIVERGED_JACOBIAN_DOMAIN will be set if you had called SNESSetJacobianDomainError(). Also take a look at the Note in https://petsc.org/release/manualpages/SNES/SNESSetFunctionDomainError/#snessetfunctiondomainerror. This provides two alternatives to simply giving up on the nonlinear solve but provides ways to either bound the tentative solutions into their physical regime or to "nudge" the tentative solutions back into their physical regime. PETSc errors are hard, meaning when one is encountered the program ends! Thus one cannot use the error handler infrastructure to recover from errors, even user created errors; that is why we provide the SNES domain "error" mechanism since it does not involve the PETSc error handlers. Good luck. > On Jul 5, 2023, at 8:54 PM, Hongrui Yu wrote: > > Hello PETSc users, > > I?d like to use the error handling mechanism to abort SNES mid-iteration if certain quantities become nonphysical by calling SETERRQ() inside FormFunction() or FormJacobian(). I?m wondering if users can define their own error handler routines to use with PetscPushErrorHandler() and PetscPopErrorHandler()? I?m also open to any suggestion on better ways to quit SNES mid-iteration. > > Thanks in advance for any help or advice! > > Hongrui -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuhongrui at utexas.edu Wed Jul 5 20:42:40 2023 From: yuhongrui at utexas.edu (Hongrui Yu) Date: Wed, 5 Jul 2023 20:42:40 -0500 Subject: [petsc-users] Abort SNES Mid-iteration In-Reply-To: References: <065d01d9afa4$7b45f560$71d1e020$@utexas.edu> Message-ID: <067301d9afab$271471e0$753d55a0$@utexas.edu> Thanks Barry, this is exactly what I was looking for! Hongrui From: Barry Smith Sent: Wednesday, July 5, 2023 8:23 PM To: Hongrui Yu Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Abort SNES Mid-iteration I assume you want to stop the SNES solve, but you do not want the program to end immediately? Instead, you want SNESSolve() to end immediately with a very useful message? This is done with https://petsc.org/release/manualpages/SNES/SNESSetFunctionDomainError/#snessetfunctiondomainerror or https://petsc.org/release/manualpages/SNES/SNESSetJacobianDomainError/#snessetjacobiandomainerror In conjunction with using this, you must call https://petsc.org/release/manualpages/SNES/SNESGetConvergedReason/#snesgetconvergedreason immediately after each SNESSolve() and check if the reason is negative. The specific values SNES_DIVERGED_FUNCTION_DOMAIN will be set if you had called SNESSetFunctionDomainError() and SNES_DIVERGED_JACOBIAN_DOMAIN will be set if you had called SNESSetJacobianDomainError(). Also take a look at the Note in https://petsc.org/release/manualpages/SNES/SNESSetFunctionDomainError/#snessetfunctiondomainerror. This provides two alternatives to simply giving up on the nonlinear solve but provides ways to either bound the tentative solutions into their physical regime or to "nudge" the tentative solutions back into their physical regime. PETSc errors are hard, meaning when one is encountered the program ends! Thus one cannot use the error handler infrastructure to recover from errors, even user created errors; that is why we provide the SNES domain "error" mechanism since it does not involve the PETSc error handlers. Good luck. On Jul 5, 2023, at 8:54 PM, Hongrui Yu > wrote: Hello PETSc users, I?d like to use the error handling mechanism to abort SNES mid-iteration if certain quantities become nonphysical by calling SETERRQ() inside FormFunction() or FormJacobian(). I?m wondering if users can define their own error handler routines to use with PetscPushErrorHandler() and PetscPopErrorHandler()? I?m also open to any suggestion on better ways to quit SNES mid-iteration. Thanks in advance for any help or advice! Hongrui -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 5 21:04:40 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 5 Jul 2023 22:04:40 -0400 Subject: [petsc-users] PETSc Discord server Message-ID: <77BA1846-5127-45CD-9C3A-26DAA5B052BF@petsc.dev> We are experimenting with using Discord as our PETSc chat space. PETSc users are welcome to join us there https://discord.gg/TAmq3kvB Barry From alexlindsay239 at gmail.com Thu Jul 6 14:09:59 2023 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Thu, 6 Jul 2023 12:09:59 -0700 Subject: [petsc-users] Nested solves and fgmres residual normalization Message-ID: Hi all, I've found that having fgmres as an outer solve and the initial residual scaling that comes with it can cause difficulties for inner solves as I'm approaching convergence, presumably because I'm running out of precision. This is the kind of thing I would normally set an absolute tolerance for, but it feels a little silly to set `-ksp_atol 1e-4` when at the beginning of solves there is no problem reaching that tolerance. Here is an example solve history. You can see that the fieldsplit_p_lsc_solve converges in a single iteration until the very end where it's clearly struggling to hit the rtol of 1e-5. Is the scaling of the fgmres residual important to the algorithm? git blame showed that the comment regarding "scale VEC_VV (the initial residual)" was last hit in 1999 (and it existed before that time), see line 125 of fgmres.c 0 KSP Residual norm 2.586864431320e-06 Residual norms for fieldsplit_p_ solve. 0 KSP Residual norm 4.692593309369e-02 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 4.048060575312e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 4.951804839158e+01 1 KSP Residual norm 9.672178594775e-05 1 KSP Residual norm 2.076409133334e-02 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.346618231753e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 8.119436471232e+01 1 KSP Residual norm 1.233237431511e-04 2 KSP Residual norm 8.753379232559e-03 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.023335686682e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.095058035930e+02 1 KSP Residual norm 8.452076098586e-05 3 KSP Residual norm 1.589938296474e-03 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 7.680981295089e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 7.768759831332e+01 1 KSP Residual norm 5.120395415891e-04 4 KSP Residual norm 4.191115394200e-04 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 8.815992679344e-07 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 5.245482238296e+01 1 KSP Residual norm 8.672309412797e-05 5 KSP Residual norm 1.053200310197e-04 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.450184043299e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 9.058043325461e+01 1 KSP Residual norm 1.643116692318e-04 6 KSP Residual norm 2.315708746597e-05 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.406236994541e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 5.158957906917e+01 1 KSP Residual norm 8.910977751230e-05 7 KSP Residual norm 2.378083073391e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 7.981173164359e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 7.271546763656e+01 1 KSP Residual norm 2.211194524633e-04 8 KSP Residual norm 8.464122393957e-07 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.627170127450e-05 2 KSP Residual norm 1.448537328052e-05 3 KSP Residual norm 1.309526985856e-05 4 KSP Residual norm 1.308667010157e-05 5 KSP Residual norm 1.254021842629e-05 6 KSP Residual norm 9.490928295823e-06 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 7.084510697783e+01 1 KSP Residual norm 1.444782053473e-04 9 KSP Residual norm 3.334507715334e-07 Linear fieldsplit_p_ solve converged due to CONVERGED_RTOL iterations 9 -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuhongrui at utexas.edu Thu Jul 6 14:38:48 2023 From: yuhongrui at utexas.edu (Hongrui Yu) Date: Thu, 6 Jul 2023 14:38:48 -0500 Subject: [petsc-users] Applying Multi-Point-Constraints with SNES Message-ID: <071501d9b041$7d296d10$777c4730$@utexas.edu> Hello PETSc users, In our Finite Element application we would like to apply Multi-Point-Constraints to some vertices. In these Multi-Point-Constraints we force certain DoF in the following nodes to be the same as those in controlling nodes. We use DMPlex to store our mesh. I'm wondering what would be a good way to implement this together with SNES? Or is there an example that does similar thing? Currently both right-hand-side and solution vectors in out application are created using DMCreateGlobalVec(). And we tried (unsuccessfully) to modify solution within FormRHS or FormJacobian routines but the solution vector seems to be read-only (for good reason). Is there a way to tell SNES about these constraints? Or perhaps one can somehow create a vector with only a subset of the DoFs (the actual "unknowns")? Thank you in advance for any help or advice! Hongrui -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Thu Jul 6 14:41:04 2023 From: ysjosh.lo at gmail.com (YuSh Lo) Date: Thu, 6 Jul 2023 14:41:04 -0500 Subject: [petsc-users] User-defined updating function in SNES Message-ID: Hi, I am using SNES to solve some nonlinear functions. After each iteration, I have an user-defined updating to do. Some values in the solution vectors must be inserted to some different location. I have tried to do it in the beginning of form RHS and form Jacobian, but the vector is read-only. The results are correct although some warnings are thrown with debug build saying that the vector is in wrong state. I also tried setting a function with SNESSetupdate, but the convergence is bad. Is the vector I get using SNESGetSolution the same as the vector that will be used in following forming Jacobian and RHS?, and is it again read only? Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 6 15:01:04 2023 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Jul 2023 16:01:04 -0400 Subject: [petsc-users] User-defined updating function in SNES In-Reply-To: References: Message-ID: On Thu, Jul 6, 2023 at 3:42?PM YuSh Lo wrote: > Hi, > > I am using SNES to solve some nonlinear functions. > After each iteration, I have an user-defined updating to do. > Some values in the solution vectors must be inserted to some different > location. > I have tried to do it in the beginning of form RHS and form Jacobian, but > the vector is read-only. The results are correct although some warnings are > thrown with debug build saying that the vector is in wrong state. I also > tried setting a function with SNESSetupdate, but the convergence is bad. > Is the vector I get using SNESGetSolution the same as the vector that will > be used in following forming Jacobian and RHS?, and is it again read only? > For these kinds of things, it might be easiest to put this in the line search. I would write a custom line search routine that did your modifications. That way you are assured that it does not short circuit anything else since the line search is where we are updating the solution directly. It should also work in the Update(), and we could try to determine what is wrong here, but that would be more digging. I think making it a line search is easier. Thanks, Matt > Thanks, > Josh > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Thu Jul 6 15:13:39 2023 From: ysjosh.lo at gmail.com (YuSh Lo) Date: Thu, 6 Jul 2023 15:13:39 -0500 Subject: [petsc-users] User-defined updating function in SNES In-Reply-To: References: Message-ID: Hi Matthew, Thank you. I will try setting up custom linesearch. Thanks, Josh Matthew Knepley ? 2023?7?6? ?? ??3:01??? > On Thu, Jul 6, 2023 at 3:42?PM YuSh Lo wrote: > >> Hi, >> >> I am using SNES to solve some nonlinear functions. >> After each iteration, I have an user-defined updating to do. >> Some values in the solution vectors must be inserted to some different >> location. >> I have tried to do it in the beginning of form RHS and form Jacobian, but >> the vector is read-only. The results are correct although some warnings are >> thrown with debug build saying that the vector is in wrong state. I also >> tried setting a function with SNESSetupdate, but the convergence is bad. >> Is the vector I get using SNESGetSolution the same as the vector that >> will be used in following forming Jacobian and RHS?, and is it again read >> only? >> > > For these kinds of things, it might be easiest to put this in the line > search. I would write a custom line search routine that did your > modifications. That way you are assured that it does not short circuit > anything else since the line search is where we are updating the solution > directly. > > It should also work in the Update(), and we could try to determine what is > wrong here, but that would be more digging. I think making it a line search > is easier. > > Thanks, > > Matt > > >> Thanks, >> Josh >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 6 15:33:37 2023 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Jul 2023 16:33:37 -0400 Subject: [petsc-users] Applying Multi-Point-Constraints with SNES In-Reply-To: <071501d9b041$7d296d10$777c4730$@utexas.edu> References: <071501d9b041$7d296d10$777c4730$@utexas.edu> Message-ID: On Thu, Jul 6, 2023 at 3:40?PM Hongrui Yu wrote: > Hello PETSc users, > > > > In our Finite Element application we would like to apply > Multi-Point-Constraints to some vertices. In these Multi-Point-Constraints > we force certain DoF in the following nodes to be the same as those in > controlling nodes. We use DMPlex to store our mesh. I?m wondering what > would be a good way to implement this together with SNES? Or is there an > example that does similar thing? > There is a simple thing that Jed implemented call "isoperiodicity" in plexsfc.c, and you can see him using it to define a periodic box in DMPlexCreateBoxMesh_Tensor_SFC_Periodicity_Private() You could use exactly this to define an affine relation between two dofs. > Currently both right-hand-side and solution vectors in out application are > created using DMCreateGlobalVec(). And we tried (unsuccessfully) to modify > solution within FormRHS or FormJacobian routines but the solution vector > seems to be read-only (for good reason). Is there a way to tell SNES about > these constraints? Or perhaps one can somehow create a vector with only a > subset of the DoFs (the actual ?unknowns?)? > Doing it as above will eliminate the constrained unknowns from the SNES automatically, so this will not be a concern. Thanks, Matt > Thank you in advance for any help or advice! > > > > Hongrui > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Thu Jul 6 19:30:24 2023 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Thu, 6 Jul 2023 17:30:24 -0700 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: References: <87cz3i7fj1.fsf@jedbrown.org> <3287ff5f-5ac1-fdff-52d1-97888568c098@itwm.fraunhofer.de> <8735479bsg.fsf@jedbrown.org> <875y7ymzc2.fsf@jedbrown.org> <15FFDCF6-48C9-4331-A9FE-932BBDD418D1@lip6.fr> Message-ID: This is an interesting article that compares a multi-level ILU algorithm to approximate commutator and augmented Lagrange methods: https://doi.org/10.1002/fld.5039 On Wed, Jun 28, 2023 at 11:37?AM Alexander Lindsay wrote: > I do believe that based off the results in > https://doi.org/10.1137/040608817 we should be able to make LSC, with > proper scaling, compare very favorably with PCD > > On Tue, Jun 27, 2023 at 10:41?AM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > >> I've opened https://gitlab.com/petsc/petsc/-/merge_requests/6642 which >> adds a couple more scaling applications of the inverse of the diagonal of A >> >> On Mon, Jun 26, 2023 at 6:06?PM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> >>> I guess that similar to the discussions about selfp, the approximation >>> of the velocity mass matrix by the diagonal of the velocity sub-matrix will >>> improve when running a transient as opposed to a steady calculation, >>> especially if the time derivative is lumped.... Just thinking while typing >>> >>> On Mon, Jun 26, 2023 at 6:03?PM Alexander Lindsay < >>> alexlindsay239 at gmail.com> wrote: >>> >>>> Returning to Sebastian's question about the correctness of the current >>>> LSC implementation: in the taxonomy paper that Jed linked to (which talks >>>> about SIMPLE, PCD, and LSC), equation 21 shows four applications of the >>>> inverse of the velocity mass matrix. In the PETSc implementation there are >>>> at most two applications of the reciprocal of the diagonal of A (an >>>> approximation to the velocity mass matrix without more plumbing, as already >>>> pointed out). It seems like for code implementations in which there are >>>> possible scaling differences between the velocity and pressure equations, >>>> that this difference in the number of inverse applications could be >>>> significant? I know Jed said that these scalings wouldn't really matter if >>>> you have a uniform grid, but I'm not 100% convinced yet. >>>> >>>> I might try fiddling around with adding two more reciprocal >>>> applications. >>>> >>>> On Fri, Jun 23, 2023 at 1:09?PM Pierre Jolivet >>>> wrote: >>>> >>>>> >>>>> On 23 Jun 2023, at 10:06 PM, Pierre Jolivet >>>>> wrote: >>>>> >>>>> >>>>> On 23 Jun 2023, at 9:39 PM, Alexander Lindsay < >>>>> alexlindsay239 at gmail.com> wrote: >>>>> >>>>> Ah, I see that if I use Pierre's new 'full' option for >>>>> -mat_schur_complement_ainv_type >>>>> >>>>> >>>>> That was not initially done by me >>>>> >>>>> >>>>> Oops, sorry for the noise, looks like it was done by me indeed >>>>> in 9399e4fd88c6621aad8fe9558ce84df37bd6fada? >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>> (though I recently tweaked MatSchurComplementComputeExplicitOperator() >>>>> a bit to use KSPMatSolve(), so that if you have a small Schur complement ? >>>>> which is not really the case for NS ? this could be a viable option, it was >>>>> previously painfully slow). >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>> that I get a single iteration for the Schur complement solve with LU. >>>>> That's a nice testing option >>>>> >>>>> On Fri, Jun 23, 2023 at 12:02?PM Alexander Lindsay < >>>>> alexlindsay239 at gmail.com> wrote: >>>>> >>>>>> I guess it is because the inverse of the diagonal form of A00 becomes >>>>>> a poor representation of the inverse of A00? I guess naively I would have >>>>>> thought that the blockdiag form of A00 is A00 >>>>>> >>>>>> On Fri, Jun 23, 2023 at 10:18?AM Alexander Lindsay < >>>>>> alexlindsay239 at gmail.com> wrote: >>>>>> >>>>>>> Hi Jed, I will come back with answers to all of your questions at >>>>>>> some point. I mostly just deal with MOOSE users who come to me and tell me >>>>>>> their solve is converging slowly, asking me how to fix it. So I generally >>>>>>> assume they have built an appropriate mesh and problem size for the problem >>>>>>> they want to solve and added appropriate turbulence modeling (although my >>>>>>> general assumption is often violated). >>>>>>> >>>>>>> > And to confirm, are you doing a nonlinearly implicit >>>>>>> velocity-pressure solve? >>>>>>> >>>>>>> Yes, this is our default. >>>>>>> >>>>>>> A general question: it seems that it is well known that the quality >>>>>>> of selfp degrades with increasing advection. Why is that? >>>>>>> >>>>>>> On Wed, Jun 7, 2023 at 8:01?PM Jed Brown wrote: >>>>>>> >>>>>>>> Alexander Lindsay writes: >>>>>>>> >>>>>>>> > This has been a great discussion to follow. Regarding >>>>>>>> > >>>>>>>> >> when time stepping, you have enough mass matrix that cheaper >>>>>>>> preconditioners are good enough >>>>>>>> > >>>>>>>> > I'm curious what some algebraic recommendations might be for high >>>>>>>> Re in >>>>>>>> > transients. >>>>>>>> >>>>>>>> What mesh aspect ratio and streamline CFL number? Assuming your >>>>>>>> model is turbulent, can you say anything about momentum thickness Reynolds >>>>>>>> number Re_?? What is your wall normal spacing in plus units? (Wall resolved >>>>>>>> or wall modeled?) >>>>>>>> >>>>>>>> And to confirm, are you doing a nonlinearly implicit >>>>>>>> velocity-pressure solve? >>>>>>>> >>>>>>>> > I've found one-level DD to be ineffective when applied >>>>>>>> monolithically or to the momentum block of a split, as it scales with the >>>>>>>> mesh size. >>>>>>>> >>>>>>>> I wouldn't put too much weight on "scaling with mesh size" per se. >>>>>>>> You want an efficient solver for the coarsest mesh that delivers sufficient >>>>>>>> accuracy in your flow regime. Constants matter. >>>>>>>> >>>>>>>> Refining the mesh while holding time steps constant changes the >>>>>>>> advective CFL number as well as cell Peclet/cell Reynolds numbers. A >>>>>>>> meaningful scaling study is to increase Reynolds number (e.g., by growing >>>>>>>> the domain) while keeping mesh size matched in terms of plus units in the >>>>>>>> viscous sublayer and Kolmogorov length in the outer boundary layer. That >>>>>>>> turns out to not be a very automatic study to do, but it's what matters and >>>>>>>> you can spend a lot of time chasing ghosts with naive scaling studies. >>>>>>>> >>>>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 6 20:20:15 2023 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 6 Jul 2023 21:20:15 -0400 Subject: [petsc-users] Nested solves and fgmres residual normalization In-Reply-To: References: Message-ID: <6C7FF319-0E41-4942-9DF1-136DC19262C7@petsc.dev> You are referring to 8 KSP Residual norm 8.464122393957e-07 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.627170127450e-05 2 KSP Residual norm 1.448537328052e-05 3 KSP Residual norm 1.309526985856e-05 4 KSP Residual norm 1.308667010157e-05 5 KSP Residual norm 1.254021842629e-05 6 KSP Residual norm 9.490928295823e-06 where because the default rtol is 1.e-5 it takes 6 iterations, mostly to get the last little bit? The thing I notice is that "Residual norms for fieldsplit_p_ solve.:" are converging pretty slowly, around 1 digit per iteration. That tells me that the inner inner solve is "oversolving", that is giving more accuracy than is likely needed. Of course, when it takes 1 iteration there is no way for you to back it off to take 1/2 an iteration :-). Some things to consider: 1) do not depend on the rtol at all but just set that inner inner solve to do a single iteration; clearly in this particular case this would work fine. (you could even turn off norm calculations in this inner solve, just use Richardson in this inner inner solve, to save a tiny bit of time), of course, only experimentation would tell if that always works for your family of problems. 2) or just use an rtol of say 1.e-3 in that inner inner solve. Likely that is ok given the relative slow convergence of the fieldsplit_p_ solve. Now if the "Residual norms for fieldsplit_p_ solve.:" was dropping say 4 or more digits each iteration, then I would not generally back off the inner inner solve since likely in that case it would not be oversolving. I don't think the scaling factor as part of the FGMRES algorithm is relevant here. Barry > On Jul 6, 2023, at 3:09 PM, Alexander Lindsay wrote: > > Hi all, I've found that having fgmres as an outer solve and the initial residual scaling that comes with it can cause difficulties for inner solves as I'm approaching convergence, presumably because I'm running out of precision. This is the kind of thing I would normally set an absolute tolerance for, but it feels a little silly to set `-ksp_atol 1e-4` when at the beginning of solves there is no problem reaching that tolerance. Here is an example solve history. You can see that the fieldsplit_p_lsc_solve converges in a single iteration until the very end where it's clearly struggling to hit the rtol of 1e-5. Is the scaling of the fgmres residual important to the algorithm? git blame showed that the comment regarding "scale VEC_VV (the initial residual)" was last hit in 1999 (and it existed before that time), see line 125 of fgmres.c > > 0 KSP Residual norm 2.586864431320e-06 > Residual norms for fieldsplit_p_ solve. > 0 KSP Residual norm 4.692593309369e-02 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 4.048060575312e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 4.951804839158e+01 > 1 KSP Residual norm 9.672178594775e-05 > 1 KSP Residual norm 2.076409133334e-02 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.346618231753e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 8.119436471232e+01 > 1 KSP Residual norm 1.233237431511e-04 > 2 KSP Residual norm 8.753379232559e-03 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.023335686682e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.095058035930e+02 > 1 KSP Residual norm 8.452076098586e-05 > 3 KSP Residual norm 1.589938296474e-03 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 7.680981295089e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.768759831332e+01 > 1 KSP Residual norm 5.120395415891e-04 > 4 KSP Residual norm 4.191115394200e-04 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 8.815992679344e-07 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 5.245482238296e+01 > 1 KSP Residual norm 8.672309412797e-05 > 5 KSP Residual norm 1.053200310197e-04 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.450184043299e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 9.058043325461e+01 > 1 KSP Residual norm 1.643116692318e-04 > 6 KSP Residual norm 2.315708746597e-05 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.406236994541e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 5.158957906917e+01 > 1 KSP Residual norm 8.910977751230e-05 > 7 KSP Residual norm 2.378083073391e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 7.981173164359e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.271546763656e+01 > 1 KSP Residual norm 2.211194524633e-04 > 8 KSP Residual norm 8.464122393957e-07 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.627170127450e-05 > 2 KSP Residual norm 1.448537328052e-05 > 3 KSP Residual norm 1.309526985856e-05 > 4 KSP Residual norm 1.308667010157e-05 > 5 KSP Residual norm 1.254021842629e-05 > 6 KSP Residual norm 9.490928295823e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.084510697783e+01 > 1 KSP Residual norm 1.444782053473e-04 > 9 KSP Residual norm 3.334507715334e-07 > Linear fieldsplit_p_ solve converged due to CONVERGED_RTOL iterations 9 From knepley at gmail.com Thu Jul 6 20:24:59 2023 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Jul 2023 21:24:59 -0400 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: References: <87cz3i7fj1.fsf@jedbrown.org> <3287ff5f-5ac1-fdff-52d1-97888568c098@itwm.fraunhofer.de> <8735479bsg.fsf@jedbrown.org> <875y7ymzc2.fsf@jedbrown.org> <15FFDCF6-48C9-4331-A9FE-932BBDD418D1@lip6.fr> Message-ID: On Thu, Jul 6, 2023 at 8:30?PM Alexander Lindsay wrote: > This is an interesting article that compares a multi-level ILU algorithm > to approximate commutator and augmented Lagrange methods: > https://doi.org/10.1002/fld.5039 > That is for incompressible NS. The results are not better than https://arxiv.org/abs/1810.03315, and that PC is considerably simpler and already implemented in PETSc. There is an update in to this https://epubs.siam.org/doi/abs/10.1137/21M1430698?casa_token=Fp_XhuZStZ0AAAAA:YDhnkW9XvAom_b8KocWz-hBEI7FAt46aw3ICa0FvCrOVCtYr9bwvtqJ4aBOTkDSvANKh6YTQEw which removes the need for complicated elements. You might need stuff like ILU for compressible flow, but I think incompressible is solved. Thanks, Matt > On Wed, Jun 28, 2023 at 11:37?AM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > >> I do believe that based off the results in >> https://doi.org/10.1137/040608817 we should be able to make LSC, with >> proper scaling, compare very favorably with PCD >> >> On Tue, Jun 27, 2023 at 10:41?AM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> >>> I've opened https://gitlab.com/petsc/petsc/-/merge_requests/6642 which >>> adds a couple more scaling applications of the inverse of the diagonal of A >>> >>> On Mon, Jun 26, 2023 at 6:06?PM Alexander Lindsay < >>> alexlindsay239 at gmail.com> wrote: >>> >>>> I guess that similar to the discussions about selfp, the approximation >>>> of the velocity mass matrix by the diagonal of the velocity sub-matrix will >>>> improve when running a transient as opposed to a steady calculation, >>>> especially if the time derivative is lumped.... Just thinking while typing >>>> >>>> On Mon, Jun 26, 2023 at 6:03?PM Alexander Lindsay < >>>> alexlindsay239 at gmail.com> wrote: >>>> >>>>> Returning to Sebastian's question about the correctness of the current >>>>> LSC implementation: in the taxonomy paper that Jed linked to (which talks >>>>> about SIMPLE, PCD, and LSC), equation 21 shows four applications of the >>>>> inverse of the velocity mass matrix. In the PETSc implementation there are >>>>> at most two applications of the reciprocal of the diagonal of A (an >>>>> approximation to the velocity mass matrix without more plumbing, as already >>>>> pointed out). It seems like for code implementations in which there are >>>>> possible scaling differences between the velocity and pressure equations, >>>>> that this difference in the number of inverse applications could be >>>>> significant? I know Jed said that these scalings wouldn't really matter if >>>>> you have a uniform grid, but I'm not 100% convinced yet. >>>>> >>>>> I might try fiddling around with adding two more reciprocal >>>>> applications. >>>>> >>>>> On Fri, Jun 23, 2023 at 1:09?PM Pierre Jolivet >>>>> wrote: >>>>> >>>>>> >>>>>> On 23 Jun 2023, at 10:06 PM, Pierre Jolivet >>>>>> wrote: >>>>>> >>>>>> >>>>>> On 23 Jun 2023, at 9:39 PM, Alexander Lindsay < >>>>>> alexlindsay239 at gmail.com> wrote: >>>>>> >>>>>> Ah, I see that if I use Pierre's new 'full' option for >>>>>> -mat_schur_complement_ainv_type >>>>>> >>>>>> >>>>>> That was not initially done by me >>>>>> >>>>>> >>>>>> Oops, sorry for the noise, looks like it was done by me indeed >>>>>> in 9399e4fd88c6621aad8fe9558ce84df37bd6fada? >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>> (though I recently >>>>>> tweaked MatSchurComplementComputeExplicitOperator() a bit to use >>>>>> KSPMatSolve(), so that if you have a small Schur complement ? which is not >>>>>> really the case for NS ? this could be a viable option, it was previously >>>>>> painfully slow). >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>> that I get a single iteration for the Schur complement solve with LU. >>>>>> That's a nice testing option >>>>>> >>>>>> On Fri, Jun 23, 2023 at 12:02?PM Alexander Lindsay < >>>>>> alexlindsay239 at gmail.com> wrote: >>>>>> >>>>>>> I guess it is because the inverse of the diagonal form of A00 >>>>>>> becomes a poor representation of the inverse of A00? I guess naively I >>>>>>> would have thought that the blockdiag form of A00 is A00 >>>>>>> >>>>>>> On Fri, Jun 23, 2023 at 10:18?AM Alexander Lindsay < >>>>>>> alexlindsay239 at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Jed, I will come back with answers to all of your questions at >>>>>>>> some point. I mostly just deal with MOOSE users who come to me and tell me >>>>>>>> their solve is converging slowly, asking me how to fix it. So I generally >>>>>>>> assume they have built an appropriate mesh and problem size for the problem >>>>>>>> they want to solve and added appropriate turbulence modeling (although my >>>>>>>> general assumption is often violated). >>>>>>>> >>>>>>>> > And to confirm, are you doing a nonlinearly implicit >>>>>>>> velocity-pressure solve? >>>>>>>> >>>>>>>> Yes, this is our default. >>>>>>>> >>>>>>>> A general question: it seems that it is well known that the quality >>>>>>>> of selfp degrades with increasing advection. Why is that? >>>>>>>> >>>>>>>> On Wed, Jun 7, 2023 at 8:01?PM Jed Brown wrote: >>>>>>>> >>>>>>>>> Alexander Lindsay writes: >>>>>>>>> >>>>>>>>> > This has been a great discussion to follow. Regarding >>>>>>>>> > >>>>>>>>> >> when time stepping, you have enough mass matrix that cheaper >>>>>>>>> preconditioners are good enough >>>>>>>>> > >>>>>>>>> > I'm curious what some algebraic recommendations might be for >>>>>>>>> high Re in >>>>>>>>> > transients. >>>>>>>>> >>>>>>>>> What mesh aspect ratio and streamline CFL number? Assuming your >>>>>>>>> model is turbulent, can you say anything about momentum thickness Reynolds >>>>>>>>> number Re_?? What is your wall normal spacing in plus units? (Wall resolved >>>>>>>>> or wall modeled?) >>>>>>>>> >>>>>>>>> And to confirm, are you doing a nonlinearly implicit >>>>>>>>> velocity-pressure solve? >>>>>>>>> >>>>>>>>> > I've found one-level DD to be ineffective when applied >>>>>>>>> monolithically or to the momentum block of a split, as it scales with the >>>>>>>>> mesh size. >>>>>>>>> >>>>>>>>> I wouldn't put too much weight on "scaling with mesh size" per se. >>>>>>>>> You want an efficient solver for the coarsest mesh that delivers sufficient >>>>>>>>> accuracy in your flow regime. Constants matter. >>>>>>>>> >>>>>>>>> Refining the mesh while holding time steps constant changes the >>>>>>>>> advective CFL number as well as cell Peclet/cell Reynolds numbers. A >>>>>>>>> meaningful scaling study is to increase Reynolds number (e.g., by growing >>>>>>>>> the domain) while keeping mesh size matched in terms of plus units in the >>>>>>>>> viscous sublayer and Kolmogorov length in the outer boundary layer. That >>>>>>>>> turns out to not be a very automatic study to do, but it's what matters and >>>>>>>>> you can spend a lot of time chasing ghosts with naive scaling studies. >>>>>>>>> >>>>>>>> >>>>>> >>>>>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 6 21:03:44 2023 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 6 Jul 2023 22:03:44 -0400 Subject: [petsc-users] User-defined updating function in SNES In-Reply-To: References: Message-ID: <3F9725BC-A8E3-408B-BBF7-BA926CB3B8D6@petsc.dev> Could you provide more context on when and why "Some values in the solution vectors must be inserted to some different location." during the solution process. SNES provides two "hooks" already that allow some "changes" to the solution during the solve process. Maybe one of the hooks is what you need, if not we may be able to provide another hook. https://petsc.org/release/manualpages/SNES/SNESLineSearchSetPreCheck/ https://petsc.org/release/manualpages/SNES/SNESLineSearchSetPostCheck/#sneslinesearchsetpostcheck It is a delicate process changing solution values within a Newton solve. > On Jul 6, 2023, at 4:13 PM, YuSh Lo wrote: > > Hi Matthew, > > Thank you. I will try setting up custom linesearch. > > Thanks, > Josh > > > Matthew Knepley > ? 2023?7?6? ?? ??3:01??? >> On Thu, Jul 6, 2023 at 3:42?PM YuSh Lo > wrote: >>> Hi, >>> >>> I am using SNES to solve some nonlinear functions. >>> After each iteration, I have an user-defined updating to do. >>> Some values in the solution vectors must be inserted to some different location. >>> I have tried to do it in the beginning of form RHS and form Jacobian, but the vector is read-only. The results are correct although some warnings are thrown with debug build saying that the vector is in wrong state. I also tried setting a function with SNESSetupdate, but the convergence is bad. >>> Is the vector I get using SNESGetSolution the same as the vector that will be used in following forming Jacobian and RHS?, and is it again read only? >> >> For these kinds of things, it might be easiest to put this in the line search. I would write a custom line search routine that did your modifications. That way you are assured that it does not short circuit anything else since the line search is where we are updating the solution directly. >> >> It should also work in the Update(), and we could try to determine what is wrong here, but that would be more digging. I think making it a line search is easier. >> >> Thanks, >> >> Matt >> >>> Thanks, >>> Josh >>> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuhongrui at utexas.edu Fri Jul 7 14:59:49 2023 From: yuhongrui at utexas.edu (Hongrui Yu) Date: Fri, 7 Jul 2023 14:59:49 -0500 Subject: [petsc-users] Applying Multi-Point-Constraints with SNES In-Reply-To: References: <071501d9b041$7d296d10$777c4730$@utexas.edu> Message-ID: <084701d9b10d$965e3d70$c31ab850$@utexas.edu> Thanks Matt. To clarify, were you referring to the following section in DMPlexCreateBoxMesh_Tensor_SFC_Periodicity_Private()? PetscCall(DMPlexSetIsoperiodicFaceSF(dm, sfper)); PetscScalar t[4][4] = {{0}}; t[0][0] = 1; t[1][1] = 1; t[2][2] = 1; t[3][3] = 1; for (PetscInt i = 0; i < dim; i++) if (periodicity[i] == DM_BOUNDARY_PERIODIC) t[i][3] = upper[i] - lower[i]; PetscCall(DMPlexSetIsoperiodicFaceTransform(dm, &t[0][0])); In my understanding, DMPlexSetIsoperiodicFaceSF() defines the points where the constrains exists. The constrains are specified in t matrix and set by calling DMPlexSetIsoperiodicFaceTransform(). Is this correct? Does constraints set by this also apply to non-displacement DoFs? Our application is of phase-field fracture in elasticity so we have a scalar DoF on each vertices in addition to displacement field. In other words, the section has two fields, where displacement field has 2 DoFs and phase-field has 1 DoF. Thanks, Hongrui From: Matthew Knepley Sent: Thursday, July 6, 2023 3:34 PM To: Hongrui Yu Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Applying Multi-Point-Constraints with SNES On Thu, Jul 6, 2023 at 3:40?PM Hongrui Yu > wrote: Hello PETSc users, In our Finite Element application we would like to apply Multi-Point-Constraints to some vertices. In these Multi-Point-Constraints we force certain DoF in the following nodes to be the same as those in controlling nodes. We use DMPlex to store our mesh. I?m wondering what would be a good way to implement this together with SNES? Or is there an example that does similar thing? There is a simple thing that Jed implemented call "isoperiodicity" in plexsfc.c, and you can see him using it to define a periodic box in DMPlexCreateBoxMesh_Tensor_SFC_Periodicity_Private() You could use exactly this to define an affine relation between two dofs. Currently both right-hand-side and solution vectors in out application are created using DMCreateGlobalVec(). And we tried (unsuccessfully) to modify solution within FormRHS or FormJacobian routines but the solution vector seems to be read-only (for good reason). Is there a way to tell SNES about these constraints? Or perhaps one can somehow create a vector with only a subset of the DoFs (the actual ?unknowns?)? Doing it as above will eliminate the constrained unknowns from the SNES automatically, so this will not be a concern. Thanks, Matt Thank you in advance for any help or advice! Hongrui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 7 15:07:56 2023 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Jul 2023 16:07:56 -0400 Subject: [petsc-users] Applying Multi-Point-Constraints with SNES In-Reply-To: <084701d9b10d$965e3d70$c31ab850$@utexas.edu> References: <071501d9b041$7d296d10$777c4730$@utexas.edu> <084701d9b10d$965e3d70$c31ab850$@utexas.edu> Message-ID: On Fri, Jul 7, 2023 at 3:59?PM Hongrui Yu wrote: > Thanks Matt. To clarify, were you referring to the following section in > DMPlexCreateBoxMesh_Tensor_SFC_Periodicity_Private()? > > > > PetscCall(DMPlexSetIsoperiodicFaceSF(dm, sfper)); > > > > PetscScalar t[4][4] = {{0}}; > > t[0][0] = 1; > > t[1][1] = 1; > > t[2][2] = 1; > > t[3][3] = 1; > > for (PetscInt i = 0; i < dim; i++) > > if (periodicity[i] == DM_BOUNDARY_PERIODIC) t[i][3] = upper[i] - > lower[i]; > > PetscCall(DMPlexSetIsoperiodicFaceTransform(dm, &t[0][0])); > > > > In my understanding, DMPlexSetIsoperiodicFaceSF() defines the points where > the constrains exists. The constrains are specified in t matrix and set by > calling DMPlexSetIsoperiodicFaceTransform(). Is this correct? > The t matrix is an affine transform to apply to the field before matching. Here it is just the identity, which is what I think you want. > Does constraints set by this also apply to non-displacement DoFs? Our > application is of phase-field fracture in elasticity so we have a scalar > DoF on each vertices in addition to displacement field. In other words, the > section has two fields, where displacement field has 2 DoFs and phase-field > has 1 DoF. > It should just work. Thanks, Matt > Thanks, > > Hongrui > > > > *From:* Matthew Knepley > *Sent:* Thursday, July 6, 2023 3:34 PM > *To:* Hongrui Yu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Applying Multi-Point-Constraints with SNES > > > > On Thu, Jul 6, 2023 at 3:40?PM Hongrui Yu wrote: > > Hello PETSc users, > > > > In our Finite Element application we would like to apply > Multi-Point-Constraints to some vertices. In these Multi-Point-Constraints > we force certain DoF in the following nodes to be the same as those in > controlling nodes. We use DMPlex to store our mesh. I?m wondering what > would be a good way to implement this together with SNES? Or is there an > example that does similar thing? > > > > There is a simple thing that Jed implemented call "isoperiodicity" in > plexsfc.c, and you can see > > him using it to define a periodic box in > > > > DMPlexCreateBoxMesh_Tensor_SFC_Periodicity_Private() > > > > You could use exactly this to define an affine relation between two dofs. > > > > Currently both right-hand-side and solution vectors in out application are > created using DMCreateGlobalVec(). And we tried (unsuccessfully) to modify > solution within FormRHS or FormJacobian routines but the solution vector > seems to be read-only (for good reason). Is there a way to tell SNES about > these constraints? Or perhaps one can somehow create a vector with only a > subset of the DoFs (the actual ?unknowns?)? > > > > Doing it as above will eliminate the constrained unknowns from the SNES > automatically, so this will not be a concern. > > > > Thanks, > > > > Matt > > > > Thank you in advance for any help or advice! > > > > Hongrui > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wutony0408 at gmail.com Mon Jul 10 15:56:51 2023 From: wutony0408 at gmail.com (Tony Wu) Date: Tue, 11 Jul 2023 04:56:51 +0800 Subject: [petsc-users] Solving a simpler PDE results in error while solving the original equation works in snes/tutorials/ex13.c Message-ID: In snes/tutorials/ex13.c, there is a function SetupPrimalProblem(), which sets up the f_0 and f_1 in PetscDSSetResidual() as described in https://petsc.org/release/manualpages/DT/PetscDSSetResidual/#petscdssetresidual , and also the boundary conditions using the exact solution trig_inhomogeneous_u(). Now, I am trying to modify f_0, f_1, and the boundary conditions in order to solve the 2D equation u+\Delta u=f, where f(x,y)=x and the boundary conditions are determined by the exact solution u(x,y)=x. In order to do this, I defined the following point-wise functions: static void f00(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const PetscScalar constants[], PetscScalar f0[]) { f0[0] -= u[0]; f0[0] += x[0]; } static void f11(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const PetscScalar constants[], PetscScalar f1[]) { PetscInt d; for (d = 0; d < dim; ++d) { f1[d] = u_x[d]; } } static void u_exact(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nc, PetscScalar *u, void *ctx) { *u = 0.0; *u += x[0]; return PETSC_SUCCESS; } I then replaced the f0, f1_u and ex in SetupPrimalProblem() by the corresponding f00, f11, and u_exact. . However, I got the following error message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./ex13 on a arch-linux-c-debug named LAPTOP by user Tue Jul 11 04:06:13 2023 [0]PETSC ERROR: Configure options --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries --with-cxx-dialect=C++11 --with-make-np=8 --download-hdf5=yes --download-mumps=yes --download-parmetis=yes --download-metis=yes --download-superlu=yes --download-hypre=yes --download-superlu_dist=yes --download-scalapack=yes --download-triangle=yes [0]PETSC ERROR: #1 DMProjectPoint_Func_Private() at /home/user/petsc/src/dm/impls/plex/plexproject.c:128 [0]PETSC ERROR: #2 DMProjectPoint_Private() at /home/user/petsc/src/dm/impls/plex/plexproject.c:395 [0]PETSC ERROR: #3 DMProjectLocal_Generic_Plex() at /home/user/petsc/src/dm/impls/plex/plexproject.c:852 [0]PETSC ERROR: #4 DMProjectFunctionLabelLocal_Plex() at /home/user/petsc/src/dm/impls/plex/plexproject.c:921 [0]PETSC ERROR: #5 DMProjectFunctionLabelLocal() at /home/user/petsc/src/dm/interface/dm.c:8071 [0]PETSC ERROR: #6 DMPlexInsertBoundaryValuesEssential() at /home/user/petsc/src/dm/impls/plex/plexfem.c:795 [0]PETSC ERROR: #7 DMPlexInsertBoundaryValues_Plex() at /home/user/petsc/src/dm/impls/plex/plexfem.c:1046 [0]PETSC ERROR: #8 DMPlexInsertBoundaryValues() at /home/user/petsc/src/dm/impls/plex/plexfem.c:1154 [0]PETSC ERROR: #9 DMPlexSNESComputeBoundaryFEM() at /home/user/petsc/src/snes/utils/dmplexsnes.c:1372 [0]PETSC ERROR: #10 SNESComputeFunction_DMLocal() at /home/user/petsc/src/snes/utils/dmlocalsnes.c:63 [0]PETSC ERROR: #11 SNESComputeFunction() at /home/user/petsc/src/snes/interface/snes.c:2389 [0]PETSC ERROR: #12 SNESSolve_NEWTONLS() at /home/user/petsc/src/snes/impls/ls/ls.c:172 [0]PETSC ERROR: #13 SNESSolve() at /home/user/petsc/src/snes/interface/snes.c:4644 [0]PETSC ERROR: #14 PetscConvEstGetConvRateSNES_Private() at /home/user/petsc/src/snes/utils/convest.c:369 [0]PETSC ERROR: #15 PetscConvEstGetConvRate() at /home/user/petsc/src/snes/utils/convest.c:468 [0]PETSC ERROR: #16 SNESSolve() at /home/user/petsc/src/snes/interface/snes.c:4561 [0]PETSC ERROR: #17 main() at ex13.c:431 [0]PETSC ERROR: Reached the main program with an out-of-range error code 584475600. This should never happen [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -convest_num_refine 2 (source: command line) [0]PETSC ERROR: -potential_petscspace_degree 1 (source: command line) [0]PETSC ERROR: -snes_convergence_estimate (source: command line) [0]PETSC ERROR: -snes_monitor (source: command line) [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF with errorcode 584475600. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- I have no idea why the program solves the original, more complicated, equation successfully while not being able to solve a simpler one. Any help or reference would be much appreciated! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 10 18:29:02 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Jul 2023 19:29:02 -0400 Subject: [petsc-users] Solving a simpler PDE results in error while solving the original equation works in snes/tutorials/ex13.c In-Reply-To: References: Message-ID: On Mon, Jul 10, 2023 at 5:02?PM Tony Wu wrote: > In snes/tutorials/ex13.c, > there is a function SetupPrimalProblem(), which sets up the f_0 and f_1 in > PetscDSSetResidual() as described in > https://petsc.org/release/manualpages/DT/PetscDSSetResidual/#petscdssetresidual > , > and also the boundary conditions using the exact solution > trig_inhomogeneous_u(). > > Now, I am trying to modify f_0, f_1, and the boundary conditions in order > to solve the 2D equation > u+\Delta u=f, > where f(x,y)=x and the boundary conditions are determined by the exact > solution u(x,y)=x. > > In order to do this, I defined the following point-wise functions: > > static void f00(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt > uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar > u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt > aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar > a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const > PetscScalar constants[], PetscScalar f0[]) > { > f0[0] -= u[0]; > f0[0] += x[0]; > } > > static void f11(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt > uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar > u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt > aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar > a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const > PetscScalar constants[], PetscScalar f1[]) > { > PetscInt d; > for (d = 0; d < dim; ++d) > { > f1[d] = u_x[d]; > } > } > > static void u_exact(PetscInt dim, PetscReal time, const PetscReal x[], > PetscInt Nc, PetscScalar *u, void *ctx) > This function is declared "void" but should be "PetscErrorCode". There must have been a warning on compile. Thanks, Matt > { > *u = 0.0; > *u += x[0]; > return PETSC_SUCCESS; > } > > I then replaced the f0, f1_u and ex in SetupPrimalProblem() > by the corresponding f00, f11, and u_exact. > . > > However, I got the following error message: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown > [0]PETSC ERROR: ./ex13 on a arch-linux-c-debug named LAPTOP by user Tue > Jul 11 04:06:13 2023 > [0]PETSC ERROR: Configure options --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 > --FOPTFLAGS=-O3 --with-shared-libraries --with-cxx-dialect=C++11 > --with-make-np=8 --download-hdf5=yes --download-mumps=yes > --download-parmetis=yes --download-metis=yes --download-superlu=yes > --download-hypre=yes --download-superlu_dist=yes --download-scalapack=yes > --download-triangle=yes > [0]PETSC ERROR: #1 DMProjectPoint_Func_Private() at > /home/user/petsc/src/dm/impls/plex/plexproject.c:128 > [0]PETSC ERROR: #2 DMProjectPoint_Private() at > /home/user/petsc/src/dm/impls/plex/plexproject.c:395 > [0]PETSC ERROR: #3 DMProjectLocal_Generic_Plex() at > /home/user/petsc/src/dm/impls/plex/plexproject.c:852 > [0]PETSC ERROR: #4 DMProjectFunctionLabelLocal_Plex() at > /home/user/petsc/src/dm/impls/plex/plexproject.c:921 > [0]PETSC ERROR: #5 DMProjectFunctionLabelLocal() at > /home/user/petsc/src/dm/interface/dm.c:8071 > [0]PETSC ERROR: #6 DMPlexInsertBoundaryValuesEssential() at > /home/user/petsc/src/dm/impls/plex/plexfem.c:795 > [0]PETSC ERROR: #7 DMPlexInsertBoundaryValues_Plex() at > /home/user/petsc/src/dm/impls/plex/plexfem.c:1046 > [0]PETSC ERROR: #8 DMPlexInsertBoundaryValues() at > /home/user/petsc/src/dm/impls/plex/plexfem.c:1154 > [0]PETSC ERROR: #9 DMPlexSNESComputeBoundaryFEM() at > /home/user/petsc/src/snes/utils/dmplexsnes.c:1372 > [0]PETSC ERROR: #10 SNESComputeFunction_DMLocal() at > /home/user/petsc/src/snes/utils/dmlocalsnes.c:63 > [0]PETSC ERROR: #11 SNESComputeFunction() at > /home/user/petsc/src/snes/interface/snes.c:2389 > [0]PETSC ERROR: #12 SNESSolve_NEWTONLS() at > /home/user/petsc/src/snes/impls/ls/ls.c:172 > [0]PETSC ERROR: #13 SNESSolve() at > /home/user/petsc/src/snes/interface/snes.c:4644 > [0]PETSC ERROR: #14 PetscConvEstGetConvRateSNES_Private() at > /home/user/petsc/src/snes/utils/convest.c:369 > [0]PETSC ERROR: #15 PetscConvEstGetConvRate() at > /home/user/petsc/src/snes/utils/convest.c:468 > [0]PETSC ERROR: #16 SNESSolve() at > /home/user/petsc/src/snes/interface/snes.c:4561 > [0]PETSC ERROR: #17 main() at ex13.c:431 > [0]PETSC ERROR: Reached the main program with an out-of-range error code > 584475600. This should never happen > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -convest_num_refine 2 (source: command line) > [0]PETSC ERROR: -potential_petscspace_degree 1 (source: command line) > [0]PETSC ERROR: -snes_convergence_estimate (source: command line) > [0]PETSC ERROR: -snes_monitor (source: command line) > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > with errorcode 584475600. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > > I have no idea why the program solves the original, more complicated, > equation successfully while not being able to solve a simpler one. > Any help or reference would be much appreciated! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Tue Jul 11 10:34:14 2023 From: liufield at gmail.com (neil liu) Date: Tue, 11 Jul 2023 11:34:14 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. Message-ID: Dear Petsc developers, My computation involves both real and complex numbers. For example, my coordinate is real, but my matrix is complex. I use VecGetArray to get the coordinates. But it seems coordinates have to be defined in a complex manner to be consistent with complex matrix. Is there some ways to work around this if I understood correctly ? Thanks, Xiaodong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 11 10:40:35 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 11 Jul 2023 11:40:35 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: References: Message-ID: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Not really. PetscScalar (and entries of vectors etc) are either real or complex, they cannot be mixed. What parts of PETSc are you using, DM? You can 1) store the coordinates in arrays and not use Vec for them. (of if you are not using DM) 2) sore them in Vec but with imaginary part 0 3) rewrite your complex systems as real systems (with twice as many unknowns, but only real ones) Barry > On Jul 11, 2023, at 11:34 AM, neil liu wrote: > > Dear Petsc developers, > > My computation involves both real and complex numbers. > For example, my coordinate is real, but my matrix is complex. > > I use VecGetArray to get the coordinates. But it seems coordinates have to be defined in a complex manner to be consistent with complex matrix. > > Is there some ways to work around this if I understood correctly ? > > Thanks, > > Xiaodong > > > From liufield at gmail.com Tue Jul 11 10:48:28 2023 From: liufield at gmail.com (neil liu) Date: Tue, 11 Jul 2023 11:48:28 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Message-ID: Thanks for your reply. I am using DMPLEX to read GMSH. And it seems I have to use DMGetCoordinatesLocal(); VecGetArray(veccords, &coords);//it seems coords has to be defined by complex either. 2) Store the coordinates in coords with imaginary part equal to zero. I got the coordinates from above function *VecGetArray*, therefore will this reduce the memory? 3) is a good way to work around that. On Tue, Jul 11, 2023 at 11:40?AM Barry Smith wrote: > > Not really. PetscScalar (and entries of vectors etc) are either real or > complex, they cannot be mixed. > > What parts of PETSc are you using, DM? > > You can > > 1) store the coordinates in arrays and not use Vec for them. (of if you > are not using DM) > 2) sore them in Vec but with imaginary part 0 > 3) rewrite your complex systems as real systems (with twice as many > unknowns, but only real ones) > > Barry > > > > On Jul 11, 2023, at 11:34 AM, neil liu wrote: > > > > Dear Petsc developers, > > > > My computation involves both real and complex numbers. > > For example, my coordinate is real, but my matrix is complex. > > > > I use VecGetArray to get the coordinates. But it seems coordinates have > to be defined in a complex manner to be consistent with complex matrix. > > > > Is there some ways to work around this if I understood correctly ? > > > > Thanks, > > > > Xiaodong > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 11 10:54:01 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Jul 2023 11:54:01 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Message-ID: On Tue, Jul 11, 2023 at 11:48?AM neil liu wrote: > Thanks for your reply. > I am using DMPLEX to read GMSH. And it seems I have to use > DMGetCoordinatesLocal(); > VecGetArray(veccords, &coords);//it seems coords has to be defined by > complex either. > > 2) Store the coordinates in coords with imaginary part equal to zero. > I got the coordinates from above function *VecGetArray*, therefore will > this reduce the memory? > > 3) is a good way to work around that. > We just store them with real part 0. Thanks, Matt > On Tue, Jul 11, 2023 at 11:40?AM Barry Smith wrote: > >> >> Not really. PetscScalar (and entries of vectors etc) are either real or >> complex, they cannot be mixed. >> >> What parts of PETSc are you using, DM? >> >> You can >> >> 1) store the coordinates in arrays and not use Vec for them. (of if you >> are not using DM) >> 2) sore them in Vec but with imaginary part 0 >> 3) rewrite your complex systems as real systems (with twice as many >> unknowns, but only real ones) >> >> Barry >> >> >> > On Jul 11, 2023, at 11:34 AM, neil liu wrote: >> > >> > Dear Petsc developers, >> > >> > My computation involves both real and complex numbers. >> > For example, my coordinate is real, but my matrix is complex. >> > >> > I use VecGetArray to get the coordinates. But it seems coordinates have >> to be defined in a complex manner to be consistent with complex matrix. >> > >> > Is there some ways to work around this if I understood correctly ? >> > >> > Thanks, >> > >> > Xiaodong >> > >> > >> > >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Tue Jul 11 11:03:37 2023 From: liufield at gmail.com (neil liu) Date: Tue, 11 Jul 2023 12:03:37 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Message-ID: Do you mean that, even if I configure with --scalar=complex, when I use vecGetarray to get the coords, the coords array is still stored in a real manner? Thanks, Xiaodong On Tue, Jul 11, 2023 at 11:54?AM Matthew Knepley wrote: > On Tue, Jul 11, 2023 at 11:48?AM neil liu wrote: > >> Thanks for your reply. >> I am using DMPLEX to read GMSH. And it seems I have to use >> DMGetCoordinatesLocal(); >> VecGetArray(veccords, &coords);//it seems coords has to be defined by >> complex either. >> >> 2) Store the coordinates in coords with imaginary part equal to zero. >> I got the coordinates from above function *VecGetArray*, therefore will >> this reduce the memory? >> >> 3) is a good way to work around that. >> > > We just store them with real part 0. > > Thanks, > > Matt > > >> On Tue, Jul 11, 2023 at 11:40?AM Barry Smith wrote: >> >>> >>> Not really. PetscScalar (and entries of vectors etc) are either real >>> or complex, they cannot be mixed. >>> >>> What parts of PETSc are you using, DM? >>> >>> You can >>> >>> 1) store the coordinates in arrays and not use Vec for them. (of if you >>> are not using DM) >>> 2) sore them in Vec but with imaginary part 0 >>> 3) rewrite your complex systems as real systems (with twice as many >>> unknowns, but only real ones) >>> >>> Barry >>> >>> >>> > On Jul 11, 2023, at 11:34 AM, neil liu wrote: >>> > >>> > Dear Petsc developers, >>> > >>> > My computation involves both real and complex numbers. >>> > For example, my coordinate is real, but my matrix is complex. >>> > >>> > I use VecGetArray to get the coordinates. But it seems coordinates >>> have to be defined in a complex manner to be consistent with complex >>> matrix. >>> > >>> > Is there some ways to work around this if I understood correctly ? >>> > >>> > Thanks, >>> > >>> > Xiaodong >>> > >>> > >>> > >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Tue Jul 11 11:16:57 2023 From: liufield at gmail.com (neil liu) Date: Tue, 11 Jul 2023 12:16:57 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Message-ID: I think I understood that. 1) In order to save memory, use a larger real system 2) vec will store the real part with imaginary part equal to zero in the context of --scalar =complex. But this will not save memory. Right? Thanks, Xiaodong On Tue, Jul 11, 2023 at 12:03?PM neil liu wrote: > Do you mean that, > > even if I configure with --scalar=complex, when I use vecGetarray to get > the coords, the coords array is still stored in a real manner? > > Thanks, > > Xiaodong > > On Tue, Jul 11, 2023 at 11:54?AM Matthew Knepley > wrote: > >> On Tue, Jul 11, 2023 at 11:48?AM neil liu wrote: >> >>> Thanks for your reply. >>> I am using DMPLEX to read GMSH. And it seems I have to use >>> DMGetCoordinatesLocal(); >>> VecGetArray(veccords, &coords);//it seems coords has to be defined by >>> complex either. >>> >>> 2) Store the coordinates in coords with imaginary part equal to zero. >>> I got the coordinates from above function *VecGetArray*, therefore will >>> this reduce the memory? >>> >>> 3) is a good way to work around that. >>> >> >> We just store them with real part 0. >> >> Thanks, >> >> Matt >> >> >>> On Tue, Jul 11, 2023 at 11:40?AM Barry Smith wrote: >>> >>>> >>>> Not really. PetscScalar (and entries of vectors etc) are either real >>>> or complex, they cannot be mixed. >>>> >>>> What parts of PETSc are you using, DM? >>>> >>>> You can >>>> >>>> 1) store the coordinates in arrays and not use Vec for them. (of if you >>>> are not using DM) >>>> 2) sore them in Vec but with imaginary part 0 >>>> 3) rewrite your complex systems as real systems (with twice as many >>>> unknowns, but only real ones) >>>> >>>> Barry >>>> >>>> >>>> > On Jul 11, 2023, at 11:34 AM, neil liu wrote: >>>> > >>>> > Dear Petsc developers, >>>> > >>>> > My computation involves both real and complex numbers. >>>> > For example, my coordinate is real, but my matrix is complex. >>>> > >>>> > I use VecGetArray to get the coordinates. But it seems coordinates >>>> have to be defined in a complex manner to be consistent with complex >>>> matrix. >>>> > >>>> > Is there some ways to work around this if I understood correctly ? >>>> > >>>> > Thanks, >>>> > >>>> > Xiaodong >>>> > >>>> > >>>> > >>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 11 11:41:45 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 11 Jul 2023 12:41:45 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> Message-ID: <93C54D75-D806-4215-985C-62F49ADAE0F9@petsc.dev> > On Jul 11, 2023, at 12:16 PM, neil liu wrote: > > I think I understood that. > > 1) In order to save memory, use a larger real system This will save a little memory, but requires a slight change to your model/code for your problem > 2) vec will store the real part with imaginary part equal to zero in the context of --scalar =complex. But this will not save memory. This will waste a little memory. Usually, coordinates do not take much of the total memory, that is solution vectors, matrices, intermediate work vectors for the solver take most of the memory so I suggest just using complex and not worrying about the few percent memory lost. > > Right? > > Thanks, > > Xiaodong > > On Tue, Jul 11, 2023 at 12:03?PM neil liu > wrote: >> Do you mean that, >> >> even if I configure with --scalar=complex, when I use vecGetarray to get the coords, the coords array is still stored in a real manner? >> >> Thanks, >> >> Xiaodong >> >> On Tue, Jul 11, 2023 at 11:54?AM Matthew Knepley > wrote: >>> On Tue, Jul 11, 2023 at 11:48?AM neil liu > wrote: >>>> Thanks for your reply. >>>> I am using DMPLEX to read GMSH. And it seems I have to use >>>> DMGetCoordinatesLocal(); >>>> VecGetArray(veccords, &coords);//it seems coords has to be defined by complex either. >>>> >>>> 2) Store the coordinates in coords with imaginary part equal to zero. >>>> I got the coordinates from above function VecGetArray, therefore will this reduce the memory? >>>> >>>> 3) is a good way to work around that. >>> >>> We just store them with real part 0. >>> >>> Thanks, >>> >>> Matt >>> >>>> On Tue, Jul 11, 2023 at 11:40?AM Barry Smith > wrote: >>>>> >>>>> Not really. PetscScalar (and entries of vectors etc) are either real or complex, they cannot be mixed. >>>>> >>>>> What parts of PETSc are you using, DM? >>>>> >>>>> You can >>>>> >>>>> 1) store the coordinates in arrays and not use Vec for them. (of if you are not using DM) >>>>> 2) sore them in Vec but with imaginary part 0 >>>>> 3) rewrite your complex systems as real systems (with twice as many unknowns, but only real ones) >>>>> >>>>> Barry >>>>> >>>>> >>>>> > On Jul 11, 2023, at 11:34 AM, neil liu > wrote: >>>>> > >>>>> > Dear Petsc developers, >>>>> > >>>>> > My computation involves both real and complex numbers. >>>>> > For example, my coordinate is real, but my matrix is complex. >>>>> > >>>>> > I use VecGetArray to get the coordinates. But it seems coordinates have to be defined in a complex manner to be consistent with complex matrix. >>>>> > >>>>> > Is there some ways to work around this if I understood correctly ? >>>>> > >>>>> > Thanks, >>>>> > >>>>> > Xiaodong >>>>> > >>>>> > >>>>> > >>>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From liufield at gmail.com Tue Jul 11 11:42:41 2023 From: liufield at gmail.com (neil liu) Date: Tue, 11 Jul 2023 12:42:41 -0400 Subject: [petsc-users] Inquiry about the PetscScalar in Petsc. In-Reply-To: <93C54D75-D806-4215-985C-62F49ADAE0F9@petsc.dev> References: <5182A336-AFD8-4DA1-A386-E7D79BED3ABD@petsc.dev> <93C54D75-D806-4215-985C-62F49ADAE0F9@petsc.dev> Message-ID: Thanks for your explanation so much. On Tue, Jul 11, 2023 at 12:41?PM Barry Smith wrote: > > > On Jul 11, 2023, at 12:16 PM, neil liu wrote: > > I think I understood that. > > 1) In order to save memory, use a larger real system > > > This will save a little memory, but requires a slight change to your > model/code for your problem > > 2) vec will store the real part with imaginary part equal to zero in the > context of --scalar =complex. But this will not save memory. > > > This will waste a little memory. > > Usually, coordinates do not take much of the total memory, that is > solution vectors, matrices, intermediate work vectors for the solver take > most of the memory > so I suggest just using complex and not worrying about the few percent > memory lost. > > > > Right? > > Thanks, > > Xiaodong > > On Tue, Jul 11, 2023 at 12:03?PM neil liu wrote: > >> Do you mean that, >> >> even if I configure with --scalar=complex, when I use vecGetarray to get >> the coords, the coords array is still stored in a real manner? >> >> Thanks, >> >> Xiaodong >> >> On Tue, Jul 11, 2023 at 11:54?AM Matthew Knepley >> wrote: >> >>> On Tue, Jul 11, 2023 at 11:48?AM neil liu wrote: >>> >>>> Thanks for your reply. >>>> I am using DMPLEX to read GMSH. And it seems I have to use >>>> DMGetCoordinatesLocal(); >>>> VecGetArray(veccords, &coords);//it seems coords has to be defined by >>>> complex either. >>>> >>>> 2) Store the coordinates in coords with imaginary part equal to zero. >>>> I got the coordinates from above function *VecGetArray*, therefore >>>> will this reduce the memory? >>>> >>>> 3) is a good way to work around that. >>>> >>> >>> We just store them with real part 0. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> On Tue, Jul 11, 2023 at 11:40?AM Barry Smith wrote: >>>> >>>>> >>>>> Not really. PetscScalar (and entries of vectors etc) are either real >>>>> or complex, they cannot be mixed. >>>>> >>>>> What parts of PETSc are you using, DM? >>>>> >>>>> You can >>>>> >>>>> 1) store the coordinates in arrays and not use Vec for them. (of if >>>>> you are not using DM) >>>>> 2) sore them in Vec but with imaginary part 0 >>>>> 3) rewrite your complex systems as real systems (with twice as many >>>>> unknowns, but only real ones) >>>>> >>>>> Barry >>>>> >>>>> >>>>> > On Jul 11, 2023, at 11:34 AM, neil liu wrote: >>>>> > >>>>> > Dear Petsc developers, >>>>> > >>>>> > My computation involves both real and complex numbers. >>>>> > For example, my coordinate is real, but my matrix is complex. >>>>> > >>>>> > I use VecGetArray to get the coordinates. But it seems coordinates >>>>> have to be defined in a complex manner to be consistent with complex >>>>> matrix. >>>>> > >>>>> > Is there some ways to work around this if I understood correctly ? >>>>> > >>>>> > Thanks, >>>>> > >>>>> > Xiaodong >>>>> > >>>>> > >>>>> > >>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From itugrik76 at mail.ru Tue Jul 11 14:55:21 2023 From: itugrik76 at mail.ru (=?UTF-8?B?0JrQvtC90YHRgtCw0L3RgtC40L0=?=) Date: Tue, 11 Jul 2023 22:55:21 +0300 Subject: [petsc-users] (no subject) Message-ID: <1689105321.125562771@f340.i.mail.ru> Hello, I'm trying to build petsc on windows. And when I make it I have such problem ? -- ?????????? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 21581 bytes Desc: not available URL: From knepley at gmail.com Tue Jul 11 15:05:50 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Jul 2023 16:05:50 -0400 Subject: [petsc-users] (no subject) In-Reply-To: <1689105321.125562771@f340.i.mail.ru> References: <1689105321.125562771@f340.i.mail.ru> Message-ID: On Tue, Jul 11, 2023 at 3:58?PM ?????????? via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, I'm trying to build petsc on windows. And when I make it I have > such problem > > Did you run configure first? Thanks, Matt > -- > ?????????? > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 21581 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Jul 11 15:09:40 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 11 Jul 2023 15:09:40 -0500 (CDT) Subject: [petsc-users] windows build In-Reply-To: <1689105321.125562771@f340.i.mail.ru> References: <1689105321.125562771@f340.i.mail.ru> Message-ID: <494bbb14-4371-7e64-c911-76786f8b5f8f@mcs.anl.gov> On Tue, 11 Jul 2023, ?????????? via petsc-users wrote: > > Hello, I'm trying to build petsc on windows. And when I make it I have such problem from the screenshot - it appears you are not using cygwin terminal(bash-shell) - as per instructions https://petsc.org/release/install/windows/ What compilers are you attempting to use with PETSc? Can you retry your build - as per the above instructions? If you still encounter issues - send us the corresponding configure.log. Also - best if you can copy/paste text from terminal - instead of screenshots Note: if you can use WSL2(linux) on your windows machine - that might be an easier install Satish From baagaard at usgs.gov Tue Jul 11 22:29:18 2023 From: baagaard at usgs.gov (Aagaard, Brad T) Date: Wed, 12 Jul 2023 03:29:18 +0000 Subject: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() Message-ID: PETSc developers, When I fail to have matching PetscFunctionBeginUser() and PetscFunctionReturn() in my code, I get segfaults and valgrind reports invalid writes at places in PETSc where memory is freed. As a result, it is difficult to track down the actual source of the error. I know there used to be a command line argument for checking for mismatches in PetscFunctionBeginUser() and PetscFunctionReturn(), but Matt said it is no longer implemented and the reporting is supposed to be automatic when debugging is turned on. This automatic reporting does not seem to be working for me. Is there a test in PETSc of this automatic reporting that I can run to verify it works in my build? Are there any additional requirements for this automatic reporting of mismatched PetscFunctionBeginUser() and PetscFunctionReturn() in my code? Must I wrap all PETSc function calls with PetscCall()? Thanks, Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 11 23:45:30 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 12 Jul 2023 00:45:30 -0400 Subject: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() In-Reply-To: References: Message-ID: #define PetscStackPop_Private(stack__, func__) \ do { \ PetscCheckAbort(!stack__.check || stack__.currentsize > 0, PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack size %d, pop %s %s:%d.\n", stack__.currentsize, func__, __FILE__, __LINE__); \ if (--stack__.currentsize < PETSCSTACKSIZE) { \ PetscCheckAbort(!stack__.check || stack__.petscroutine[stack__.currentsize] != 1 || stack__.function[stack__.currentsize] == (const char *)(func__), PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack: push from %s %s:%d. Pop from %s %s:%d.\n", \ stack__.function[stack__.currentsize], stack__.file[stack__.currentsize], stack__.line[stack__.currentsize], func__, __FILE__, __LINE__); \ It is checking on each pop (return) that the current function it is in is the same function that the begin was called in. You do not have to call through PetscCall() to have this stuff work (though we strong recommend adding them). Even if you have a return and no outstanding begins it should not crash Maybe you need to see a couple of crashes so we try to figure out what is going on. > On Jul 11, 2023, at 11:29 PM, Aagaard, Brad T via petsc-users wrote: > > PETSc developers, > > When I fail to have matching PetscFunctionBeginUser() and PetscFunctionReturn() in my code, I get segfaults and valgrind reports invalid writes at places in PETSc where memory is freed. As a result, it is difficult to track down the actual source of the error. I know there used to be a command line argument for checking for mismatches in PetscFunctionBeginUser() and PetscFunctionReturn(), but Matt said it is no longer implemented and the reporting is supposed to be automatic when debugging is turned on. This automatic reporting does not seem to be working for me. > > Is there a test in PETSc of this automatic reporting that I can run to verify it works in my build? > > Are there any additional requirements for this automatic reporting of mismatched PetscFunctionBeginUser() and PetscFunctionReturn() in my code? Must I wrap all PETSc function calls with PetscCall()? > > Thanks, > Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsfaraway at gmail.com Wed Jul 12 03:18:57 2023 From: jsfaraway at gmail.com (Runfeng Jin) Date: Wed, 12 Jul 2023 16:18:57 +0800 Subject: [petsc-users] [SLEPc] With the increasing of processors, result change in 1e-5 and solving time increase. krylovschur for MATSBAIJ matrix's smallest eigenvalue Message-ID: Hi, When I try to increase the number of processors to solve the same matrix(to acquire the smallest eigenvalue) , I find all the results differ from each other within the 1e-5 scope (Though the ||Ax-kx||/||kx|| are all achieve 1e-8) . And the solve time are first decreasing then increasing. my question is (1) Is there anyway to make the result more constant (2) Does the time decreasing because the matrix dimension are too small for so many processors? Is there any way to reduce the solve time when increasing the number of processors? Thank you! Runfeng matrix type MATSBAIJ matrix dimension 2078802 solver krylovschur blocksize 1 dimension of the subspace PETSC_DEFAULT number of eigenvalues 6 the maximum dimension allowed for the projected problem PETSC_DEFAULT -eps_non_hermitian number of processors result solve time 16 -302.06881196 526 32 -302.06881892 224 64 -302.06881989 139 128 -302.06881236 122 256 -302.06881938 285 512 -302.06881029 510 640 -302.06882377 291 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Jul 12 03:37:09 2023 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 12 Jul 2023 10:37:09 +0200 Subject: [petsc-users] [SLEPc] With the increasing of processors, result change in 1e-5 and solving time increase. krylovschur for MATSBAIJ matrix's smallest eigenvalue In-Reply-To: References: Message-ID: <7DCC963E-3647-4687-9CA3-84953C8BD099@dsic.upv.es> The computed eigenvalue has 7 matching digits, which agrees with the used tolerance. If you want more matching digits you have to reduce the tolerance. The performance seems reasonable for up to 64 processes, so yes the problem may be too small for more processes. But performance depends also a lot on the sparsity pattern of the matrix. Take into account that scalability of matrix-vector products in SBAIJ matrices is expected to be worse than in AIJ. Anyway, to answer questions about performance it is better that you send the output of -log_view Jose > El 12 jul 2023, a las 10:18, Runfeng Jin escribi?: > > Hi, > When I try to increase the number of processors to solve the same matrix(to acquire the smallest eigenvalue) , I find all the results differ from each other within the 1e-5 scope (Though the ||Ax-kx||/||kx|| are all achieve 1e-8) . And the solve time are first decreasing then increasing. > my question is > (1) Is there anyway to make the result more constant > (2) Does the time decreasing because the matrix dimension are too small for so many processors? Is there any way to reduce the solve time when increasing the number of processors? > > Thank you! > Runfeng > > > matrix type MATSBAIJ > matrix dimension 2078802 > solver krylovschur > blocksize 1 > dimension of the subspace PETSC_DEFAULT > number of eigenvalues 6 > the maximum dimension allowed for the projected problem PETSC_DEFAULT > -eps_non_hermitian > > number of processors result solve time > 16 -302.06881196 526 > 32 -302.06881892 224 > 64 -302.06881989 139 > 128 -302.06881236 122 > 256 -302.06881938 285 > 512 -302.06881029 510 > 640 -302.06882377 291 From nicolas.tardieu at edf.fr Wed Jul 12 11:04:09 2023 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Wed, 12 Jul 2023 16:04:09 +0000 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py Message-ID: Dear PETSc team, In the attached example, I set up a block pc for a saddle-point problem in petsc4py. The IS define the unknowns, namely some physical quantity (phys) and a Lagrange multiplier (lags). I would like to attach a near null space to the physical block, in order to get the best performance from an AMG pc. I have been trying hard, attaching it to the initial block, to the IS but no matter what I am doing, when it comes to "ksp_view", no near null space is attached to the matrix. Could you please help me figure out what I am doing wrong ? Thanks, Nicolas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- A non-text attachment was scrubbed... Name: test.tgz Type: application/x-compressed-tar Size: 38332 bytes Desc: test.tgz URL: From quentin.chevalier at polytechnique.edu Wed Jul 12 12:04:24 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Wed, 12 Jul 2023 19:04:24 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem Message-ID: Hello PETSc Users, I have a generalised eigenvalue problem : Ax= lambda Bx I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. I use PETSc in complex mode through the petsc4py bridge. Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. Thank you for your time, Quentin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Jul 12 12:24:14 2023 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 12 Jul 2023 19:24:14 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: Message-ID: By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. Jose > El 12 jul 2023, a las 19:04, Quentin Chevalier escribi?: > > Hello PETSc Users, > > I have a generalised eigenvalue problem : Ax= lambda Bx > I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > I use PETSc in complex mode through the petsc4py bridge. > > Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > Thank you for your time, > > Quentin From baagaard at usgs.gov Wed Jul 12 12:30:59 2023 From: baagaard at usgs.gov (Aagaard, Brad T) Date: Wed, 12 Jul 2023 17:30:59 +0000 Subject: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() In-Reply-To: References: Message-ID: I created a small toy example (attached) that suggests that the verification of matching PetscFunctionBeginUser() and PetscFunctionReturn() fails when PetscFunctionReturn() is missing or in some cases when different functions are missing PetscFunctionBeginUser() or PetscFunctionReturn(). The cases where the code crashes are much more complex and not easily reproduced in a small toy example. Here is the output for 4 cases of running my toy example: CASE 1: Correct stack. No errors. ./check_petscstack Testing correct stack... Layer 0: Calling layer 1 Layer 1: Noop CASE 2: Missing PetscFunctionBeginUser() in layer 0 is correctly detected and error message is generated. ./check_petscstack missing_begin Testing missing PetscFunctionBeginUser() in layer 0... Layer 0: Calling layer 1 Layer 1: Noop [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Invalid stack size 0, pop layer0 /home/baagaard/src/cig/pylith/tests/libtests/utils/TestPetscStack.cc:54. [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.19.2-475-g71252c2aa25 GIT Date: 2023-06-15 13:06:55 -0600 [0]PETSC ERROR: /home/baagaard/scratch/build/gcc-9.3/cig/pylith-debug/tests/libtests/utils/.libs/check_petscstack on a arch-gcc-9.3_debug named igskcicguwarlng by baagaard Wed Jul 12 11:20:50 2023 [0]PETSC ERROR: Configure options --PETSC_ARCH=arch-gcc-9.3_debug --with-debugging=1 --with-clanguage=c --with-mpi-compilers=1 --with-shared-libraries=1 --with-64-bit-points=1 --with-large-file-io=1 --with-lgrind=0 --download-chaco=1 --download-parmetis=1 --download-metis=1 --download-triangle --download-ml=1 --download-superlu=1 --with-fc=0 --download-f2cblaslapack --with-hdf5=1 --with-hdf5-dir=/software/baagaard/hdf5-1.12.1/gcc-9.3 --with-zlib=1 CFLAGS=-g [0]PETSC ERROR: #1 layer0() at /home/baagaard/src/cig/pylith/tests/libtests/utils/TestPetscStack.cc:54 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF with errorcode 77. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- CASE 3: Missing PetscFunctionReturn() in layer 1 is not detected. ./check_petscstack missing_return Testing missing PetscFunctionReturn() in layer 1... Layer 0: Calling layer 1 Layer 1: Noop CASE 4: Missing PetscFunctionBeginUser() in layer 0 and missing PetscFunctionReturn() in layer 1 are not detected. ./check_petscstack missing_both Testing missing PetsFunctionBeginUser() in layer 0 and PetscFunctionReturn() in layer 1... Layer 0: Calling layer 1 Layer 1: Noop ________________________________________ From: Barry Smith Sent: Tuesday, July 11, 2023 10:45 PM To: Aagaard, Brad T Cc: petsc-users at mcs.anl.gov Subject: [EXTERNAL] Re: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding. #define PetscStackPop_Private(stack__, func__) \ do { \ PetscCheckAbort(!stack__.check || stack__.currentsize > 0, PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack size %d, pop %s %s:%d.\n", stack__.currentsize, func__, __FILE__, __LINE__); \ if (--stack__.currentsize < PETSCSTACKSIZE) { \ PetscCheckAbort(!stack__.check || stack__.petscroutine[stack__.currentsize] != 1 || stack__.function[stack__.currentsize] == (const char *)(func__), PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack: push from %s %s:%d. Pop from %s %s:%d.\n", \ stack__.function[stack__.currentsize], stack__.file[stack__.currentsize], stack__.line[stack__.currentsize], func__, __FILE__, __LINE__); \ It is checking on each pop (return) that the current function it is in is the same function that the begin was called in. You do not have to call through PetscCall() to have this stuff work (though we strong recommend adding them). Even if you have a return and no outstanding begins it should not crash Maybe you need to see a couple of crashes so we try to figure out what is going on. On Jul 11, 2023, at 11:29 PM, Aagaard, Brad T via petsc-users wrote: PETSc developers, When I fail to have matching PetscFunctionBeginUser() and PetscFunctionReturn() in my code, I get segfaults and valgrind reports invalid writes at places in PETSc where memory is freed. As a result, it is difficult to track down the actual source of the error. I know there used to be a command line argument for checking for mismatches in PetscFunctionBeginUser() and PetscFunctionReturn(), but Matt said it is no longer implemented and the reporting is supposed to be automatic when debugging is turned on. This automatic reporting does not seem to be working for me. Is there a test in PETSc of this automatic reporting that I can run to verify it works in my build? Are there any additional requirements for this automatic reporting of mismatched PetscFunctionBeginUser() and PetscFunctionReturn() in my code? Must I wrap all PETSc function calls with PetscCall()? Thanks, Brad -------------- next part -------------- A non-text attachment was scrubbed... Name: TestPetscStack.cc Type: text/x-c++src Size: 2018 bytes Desc: TestPetscStack.cc URL: From pierre.jolivet at lip6.fr Wed Jul 12 12:52:21 2023 From: pierre.jolivet at lip6.fr (Pierre Jolivet) Date: Wed, 12 Jul 2023 19:52:21 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: Message-ID: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users wrote: > > Dear PETSc team, > > In the attached example, I set up a block pc for a saddle-point problem in petsc4py. The IS define the unknowns, namely some physical quantity (phys) and a Lagrange multiplier (lags). > I would like to attach a near null space to the physical block, in order to get the best performance from an AMG pc. > I have been trying hard, attaching it to the initial block, to the IS but no matter what I am doing, when it comes to "ksp_view", no near null space is attached to the matrix. > > Could you please help me figure out what I am doing wrong ? Are you using a double-precision 32-bit integers real build of PETSc? Is it --with-debugging=0? Because with my debug build, I get the following error (thus explaining why it?s not attached to the KSP). Traceback (most recent call last): File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in ns = NullSpace().create(True, [v], comm=comm) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "petsc4py/PETSc/Mat.pyx", line 5611, in petsc4py.PETSc.NullSpace.create petsc4py.PETSc.Error: error code 62 [0] MatNullSpaceCreate() at /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 [0] Invalid argument [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 Furthermore, if you set yourself the constant vector in the near null-space, then the first argument of create() must be False, otherwise, you?ll have twice the same vector, and you?ll end up with another error (the vectors in the near null-space must be orthonormal). If things still don?t work after those couple of fixes, please feel free to send an up-to-date reproducer. Thanks, Pierre > Thanks, > Nicolas > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > From bsmith at petsc.dev Wed Jul 12 13:35:51 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 12 Jul 2023 14:35:51 -0400 Subject: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() In-Reply-To: References: Message-ID: <3B04DBE0-6CA8-44AF-8277-07829DC353A7@petsc.dev> Ok, I don't understand the logic behind this but if you use PetscFunctionBeginUser (not PetscFunctionBegin) the check on the exit to the function is not done. PetscCheckAbort(!stack__.check || stack__.petscroutine[stack__.currentsize] != 1 || stack__.function[stack__.currentsize] == (const char *)(func__), PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack: push from %s %s:%d. Pop from %s %s:%d.\n", \ since petscroutine[] has been set to 2 the middle || is true and so PetscCheckAbort() is not triggered. Jacob, do you remember the logic behind skipping the checks here for user routines? Seems like it should check it, if the user buys into trying to use PetscFunctionBeginUser/PetscFunctionReturn we should have checks to make sure they are doing it correctly. Barry > On Jul 12, 2023, at 1:30 PM, Aagaard, Brad T wrote: > > I created a small toy example (attached) that suggests that the verification of matching PetscFunctionBeginUser() and PetscFunctionReturn() fails when PetscFunctionReturn() is missing or in some cases when different functions are missing PetscFunctionBeginUser() or PetscFunctionReturn(). The cases where the code crashes are much more complex and not easily reproduced in a small toy example. > > Here is the output for 4 cases of running my toy example: > > CASE 1: Correct stack. No errors. > > ./check_petscstack > Testing correct stack... > Layer 0: Calling layer 1 > Layer 1: Noop > > CASE 2: Missing PetscFunctionBeginUser() in layer 0 is correctly detected and error message is generated. > > ./check_petscstack missing_begin > Testing missing PetscFunctionBeginUser() in layer 0... > Layer 0: Calling layer 1 > Layer 1: Noop > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Invalid stack size 0, pop layer0 /home/baagaard/src/cig/pylith/tests/libtests/utils/TestPetscStack.cc:54. > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.19.2-475-g71252c2aa25 GIT Date: 2023-06-15 13:06:55 -0600 > [0]PETSC ERROR: /home/baagaard/scratch/build/gcc-9.3/cig/pylith-debug/tests/libtests/utils/.libs/check_petscstack on a arch-gcc-9.3_debug named igskcicguwarlng by baagaard Wed Jul 12 11:20:50 2023 > [0]PETSC ERROR: Configure options --PETSC_ARCH=arch-gcc-9.3_debug --with-debugging=1 --with-clanguage=c --with-mpi-compilers=1 --with-shared-libraries=1 --with-64-bit-points=1 --with-large-file-io=1 --with-lgrind=0 --download-chaco=1 --download-parmetis=1 --download-metis=1 --download-triangle --download-ml=1 --download-superlu=1 --with-fc=0 --download-f2cblaslapack --with-hdf5=1 --with-hdf5-dir=/software/baagaard/hdf5-1.12.1/gcc-9.3 --with-zlib=1 CFLAGS=-g > [0]PETSC ERROR: #1 layer0() at /home/baagaard/src/cig/pylith/tests/libtests/utils/TestPetscStack.cc:54 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF > with errorcode 77. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > CASE 3: Missing PetscFunctionReturn() in layer 1 is not detected. > > ./check_petscstack missing_return > Testing missing PetscFunctionReturn() in layer 1... > Layer 0: Calling layer 1 > Layer 1: Noop > > CASE 4: Missing PetscFunctionBeginUser() in layer 0 and missing PetscFunctionReturn() in layer 1 are not detected. > > ./check_petscstack missing_both > Testing missing PetsFunctionBeginUser() in layer 0 and PetscFunctionReturn() in layer 1... > Layer 0: Calling layer 1 > Layer 1: Noop > > ________________________________________ > From: Barry Smith > Sent: Tuesday, July 11, 2023 10:45 PM > To: Aagaard, Brad T > Cc: petsc-users at mcs.anl.gov > Subject: [EXTERNAL] Re: [petsc-users] Verifying matching PetscFunctionBeginUser() and PetscFunctionReturn() > > > > This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding. > > > > #define PetscStackPop_Private(stack__, func__) \ > do { \ > PetscCheckAbort(!stack__.check || stack__.currentsize > 0, PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack size %d, pop %s %s:%d.\n", stack__.currentsize, func__, __FILE__, __LINE__); \ > if (--stack__.currentsize < PETSCSTACKSIZE) { \ > PetscCheckAbort(!stack__.check || stack__.petscroutine[stack__.currentsize] != 1 || stack__.function[stack__.currentsize] == (const char *)(func__), PETSC_COMM_SELF, PETSC_ERR_PLIB, "Invalid stack: push from %s %s:%d. Pop from %s %s:%d.\n", \ > stack__.function[stack__.currentsize], stack__.file[stack__.currentsize], stack__.line[stack__.currentsize], func__, __FILE__, __LINE__); \ > > > It is checking on each pop (return) that the current function it is in is the same function that the begin was called in. > > You do not have to call through PetscCall() to have this stuff work (though we strong recommend adding them). > > Even if you have a return and no outstanding begins it should not crash > > Maybe you need to see a couple of crashes so we try to figure out what is going on. > > > On Jul 11, 2023, at 11:29 PM, Aagaard, Brad T via petsc-users wrote: > > PETSc developers, > > When I fail to have matching PetscFunctionBeginUser() and PetscFunctionReturn() in my code, I get segfaults and valgrind reports invalid writes at places in PETSc where memory is freed. As a result, it is difficult to track down the actual source of the error. I know there used to be a command line argument for checking for mismatches in PetscFunctionBeginUser() and PetscFunctionReturn(), but Matt said it is no longer implemented and the reporting is supposed to be automatic when debugging is turned on. This automatic reporting does not seem to be working for me. > > Is there a test in PETSc of this automatic reporting that I can run to verify it works in my build? > > Are there any additional requirements for this automatic reporting of mismatched PetscFunctionBeginUser() and PetscFunctionReturn() in my code? Must I wrap all PETSc function calls with PetscCall()? > > Thanks, > Brad > > From nicolas.tardieu at edf.fr Thu Jul 13 03:17:48 2023 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Thu, 13 Jul 2023 08:17:48 +0000 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: Dear Pierre, You are absolutely right. I was using a --with-debugging=0 (aka release) install and this is definitely an error. Once I used my debug install, I found the way to fix my problem. The solution is in the attached script: I first need to extract the correct block from the PC operator's MatNest and then append the null space to it. Anyway this is a bit tricky... Regards, Nicolas ________________________________ De : pierre.jolivet at lip6.fr Envoy? : mercredi 12 juillet 2023 19:52 ? : TARDIEU Nicolas Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] Near null space for a fieldsplit in petsc4py > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users wrote: > > Dear PETSc team, > > In the attached example, I set up a block pc for a saddle-point problem in petsc4py. The IS define the unknowns, namely some physical quantity (phys) and a Lagrange multiplier (lags). > I would like to attach a near null space to the physical block, in order to get the best performance from an AMG pc. > I have been trying hard, attaching it to the initial block, to the IS but no matter what I am doing, when it comes to "ksp_view", no near null space is attached to the matrix. > > Could you please help me figure out what I am doing wrong ? Are you using a double-precision 32-bit integers real build of PETSc? Is it --with-debugging=0? Because with my debug build, I get the following error (thus explaining why it?s not attached to the KSP). Traceback (most recent call last): File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in ns = NullSpace().create(True, [v], comm=comm) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "petsc4py/PETSc/Mat.pyx", line 5611, in petsc4py.PETSc.NullSpace.create petsc4py.PETSc.Error: error code 62 [0] MatNullSpaceCreate() at /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 [0] Invalid argument [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 Furthermore, if you set yourself the constant vector in the near null-space, then the first argument of create() must be False, otherwise, you?ll have twice the same vector, and you?ll end up with another error (the vectors in the near null-space must be orthonormal). If things still don?t work after those couple of fixes, please feel free to send an up-to-date reproducer. Thanks, Pierre > Thanks, > Nicolas > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_NullSpace.py.gz Type: application/gzip Size: 1442 bytes Desc: test_NullSpace.py.gz URL: From pierre.jolivet at lip6.fr Thu Jul 13 04:33:17 2023 From: pierre.jolivet at lip6.fr (Pierre Jolivet) Date: Thu, 13 Jul 2023 11:33:17 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: Dear Nicolas, > On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas wrote: > > Dear Pierre, > > You are absolutely right. I was using a --with-debugging=0 (aka release) install and this is definitely an error. > Once I used my debug install, I found the way to fix my problem. The solution is in the attached script: I first need to extract the correct block from the PC operator's MatNest and then append the null space to it. > Anyway this is a bit tricky... Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, you name it. You first need the initial PCSetUp() so that the bare minimum is put in place, then you have to fetch things yourself and adapt it to your needs. We had a similar discussion with the MEF++ people last week, there is currently no way around this, AFAIK. Thanks, Pierre > Regards, > Nicolas > > De : pierre.jolivet at lip6.fr > Envoy? : mercredi 12 juillet 2023 19:52 > ? : TARDIEU Nicolas > Cc : petsc-users at mcs.anl.gov > Objet : Re: [petsc-users] Near null space for a fieldsplit in petsc4py > > > > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users wrote: > > > > Dear PETSc team, > > > > In the attached example, I set up a block pc for a saddle-point problem in petsc4py. The IS define the unknowns, namely some physical quantity (phys) and a Lagrange multiplier (lags). > > I would like to attach a near null space to the physical block, in order to get the best performance from an AMG pc. > > I have been trying hard, attaching it to the initial block, to the IS but no matter what I am doing, when it comes to "ksp_view", no near null space is attached to the matrix. > > > > Could you please help me figure out what I am doing wrong ? > > Are you using a double-precision 32-bit integers real build of PETSc? > Is it --with-debugging=0? > Because with my debug build, I get the following error (thus explaining why it?s not attached to the KSP). > Traceback (most recent call last): > File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in > ns = NullSpace().create(True, [v], comm=comm) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "petsc4py/PETSc/Mat.pyx", line 5611, in petsc4py.PETSc.NullSpace.create > petsc4py.PETSc.Error: error code 62 > [0] MatNullSpaceCreate() at /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 > [0] Invalid argument > [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 > > Furthermore, if you set yourself the constant vector in the near null-space, then the first argument of create() must be False, otherwise, you?ll have twice the same vector, and you?ll end up with another error (the vectors in the near null-space must be orthonormal). > If things still don?t work after those couple of fixes, please feel free to send an up-to-date reproducer. > > Thanks, > Pierre > > > Thanks, > > Nicolas > > > > > > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > > ____________________________________________________ > > > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 13 06:44:04 2023 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Jul 2023 07:44:04 -0400 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: On Thu, Jul 13, 2023 at 5:33?AM Pierre Jolivet wrote: > Dear Nicolas, > > On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas > wrote: > > Dear Pierre, > > You are absolutely right. I was using a --with-debugging=0 (aka release) > install and this is definitely an error. > Once I used my debug install, I found the way to fix my problem. The > solution is in the attached script: I first need to extract the correct > block from the PC operator's MatNest and then append the null space to it. > Anyway this is a bit tricky... > > > Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, you > name it. > You first need the initial PCSetUp() so that the bare minimum is put in > place, then you have to fetch things yourself and adapt it to your needs. > We had a similar discussion with the MEF++ people last week, there is > currently no way around this, AFAIK. > Actually, I hated this as well, so I built a way around it _if_ you are using a DM to define the problem. Then you can set a "nullspace constructor" to make it if the field you are talking about is ever extracted. You use DMSetNullSpaceConstructor(). I do this in SNES ex62 and ex69, and other examples. Thanks, Matt > Thanks, > Pierre > > Regards, > Nicolas > > ------------------------------ > *De :* pierre.jolivet at lip6.fr > *Envoy? :* mercredi 12 juillet 2023 19:52 > *? :* TARDIEU Nicolas > *Cc :* petsc-users at mcs.anl.gov > *Objet :* Re: [petsc-users] Near null space for a fieldsplit in petsc4py > > > > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Dear PETSc team, > > > > In the attached example, I set up a block pc for a saddle-point problem > in petsc4py. The IS define the unknowns, namely some physical quantity > (phys) and a Lagrange multiplier (lags). > > I would like to attach a near null space to the physical block, in order > to get the best performance from an AMG pc. > > I have been trying hard, attaching it to the initial block, to the IS > but no matter what I am doing, when it comes to "ksp_view", no near null > space is attached to the matrix. > > > > Could you please help me figure out what I am doing wrong ? > > Are you using a double-precision 32-bit integers real build of PETSc? > Is it --with-debugging=0? > Because with my debug build, I get the following error (thus explaining > why it?s not attached to the KSP). > Traceback (most recent call last): > File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in > > ns = NullSpace().create(True, [v], comm=comm) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "petsc4py/PETSc/Mat.pyx", line 5611, in > petsc4py.PETSc.NullSpace.create > petsc4py.PETSc.Error: error code 62 > [0] MatNullSpaceCreate() at > /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 > [0] Invalid argument > [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 > > Furthermore, if you set yourself the constant vector in the near > null-space, then the first argument of create() must be False, otherwise, > you?ll have twice the same vector, and you?ll end up with another error > (the vectors in the near null-space must be orthonormal). > If things still don?t work after those couple of fixes, please feel free > to send an up-to-date reproducer. > > Thanks, > Pierre > > > Thanks, > > Nicolas > > > > > > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit > de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > > ____________________________________________________ > > > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From niko.karin at gmail.com Thu Jul 13 11:22:54 2023 From: niko.karin at gmail.com (Karin&NiKo) Date: Thu, 13 Jul 2023 18:22:54 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: Thank you very much for the information Matt. Unfortunately, I do not use DM.... :-( Le jeu. 13 juil. 2023 ? 13:44, Matthew Knepley a ?crit : > On Thu, Jul 13, 2023 at 5:33?AM Pierre Jolivet > wrote: > >> Dear Nicolas, >> >> On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas >> wrote: >> >> Dear Pierre, >> >> You are absolutely right. I was using a --with-debugging=0 (aka release) >> install and this is definitely an error. >> Once I used my debug install, I found the way to fix my problem. The >> solution is in the attached script: I first need to extract the correct >> block from the PC operator's MatNest and then append the null space to it. >> Anyway this is a bit tricky... >> >> >> Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, you >> name it. >> You first need the initial PCSetUp() so that the bare minimum is put in >> place, then you have to fetch things yourself and adapt it to your needs. >> We had a similar discussion with the MEF++ people last week, there is >> currently no way around this, AFAIK. >> > > Actually, I hated this as well, so I built a way around it _if_ you are > using a DM to define the problem. Then > you can set a "nullspace constructor" to make it if the field you are > talking about is ever extracted. You use DMSetNullSpaceConstructor(). I do > this in SNES ex62 and ex69, and other examples. > > Thanks, > > Matt > > >> Thanks, >> Pierre >> >> Regards, >> Nicolas >> >> ------------------------------ >> *De :* pierre.jolivet at lip6.fr >> *Envoy? :* mercredi 12 juillet 2023 19:52 >> *? :* TARDIEU Nicolas >> *Cc :* petsc-users at mcs.anl.gov >> *Objet :* Re: [petsc-users] Near null space for a fieldsplit in petsc4py >> >> >> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users < >> petsc-users at mcs.anl.gov> wrote: >> > >> > Dear PETSc team, >> > >> > In the attached example, I set up a block pc for a saddle-point problem >> in petsc4py. The IS define the unknowns, namely some physical quantity >> (phys) and a Lagrange multiplier (lags). >> > I would like to attach a near null space to the physical block, in >> order to get the best performance from an AMG pc. >> > I have been trying hard, attaching it to the initial block, to the IS >> but no matter what I am doing, when it comes to "ksp_view", no near null >> space is attached to the matrix. >> > >> > Could you please help me figure out what I am doing wrong ? >> >> Are you using a double-precision 32-bit integers real build of PETSc? >> Is it --with-debugging=0? >> Because with my debug build, I get the following error (thus explaining >> why it?s not attached to the KSP). >> Traceback (most recent call last): >> File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in >> >> ns = NullSpace().create(True, [v], comm=comm) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "petsc4py/PETSc/Mat.pyx", line 5611, in >> petsc4py.PETSc.NullSpace.create >> petsc4py.PETSc.Error: error code 62 >> [0] MatNullSpaceCreate() at >> /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 >> [0] Invalid argument >> [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 >> >> Furthermore, if you set yourself the constant vector in the near >> null-space, then the first argument of create() must be False, otherwise, >> you?ll have twice the same vector, and you?ll end up with another error >> (the vectors in the near null-space must be orthonormal). >> If things still don?t work after those couple of fixes, please feel free >> to send an up-to-date reproducer. >> >> Thanks, >> Pierre >> >> > Thanks, >> > Nicolas >> > >> > >> > >> > >> > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >> ?tablis ? l'intention exclusive des destinataires et les informations qui y >> figurent sont strictement confidentielles. Toute utilisation de ce Message >> non conforme ? sa destination, toute diffusion ou toute publication totale >> ou partielle, est interdite sauf autorisation expresse. >> > >> > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit >> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou >> partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de >> votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace >> sur quelque support que ce soit. Nous vous remercions ?galement d'en >> avertir imm?diatement l'exp?diteur par retour du message. >> > >> > Il est impossible de garantir que les communications par messagerie >> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >> erreur ou virus. >> > ____________________________________________________ >> > >> > This message and any attachments (the 'Message') are intended solely >> for the addressees. The information contained in this Message is >> confidential. Any use of information contained in this Message not in >> accord with its purpose, any dissemination or disclosure, either whole or >> partial, is prohibited except formal approval. >> > >> > If you are not the addressee, you may not copy, forward, disclose or >> use any part of it. If you have received this message in error, please >> delete it and all copies from your system and notify the sender immediately >> by return message. >> > >> > E-mail communication cannot be guaranteed to be timely secure, error or >> virus-free. >> > >> >> >> Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >> ?tablis ? l'intention exclusive des destinataires et les informations qui y >> figurent sont strictement confidentielles. Toute utilisation de ce Message >> non conforme ? sa destination, toute diffusion ou toute publication totale >> ou partielle, est interdite sauf autorisation expresse. >> Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de >> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou >> partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de >> votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace >> sur quelque support que ce soit. Nous vous remercions ?galement d'en >> avertir imm?diatement l'exp?diteur par retour du message. >> Il est impossible de garantir que les communications par messagerie >> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >> erreur ou virus. >> ____________________________________________________ >> This message and any attachments (the 'Message') are intended solely for >> the addressees. The information contained in this Message is confidential. >> Any use of information contained in this Message not in accord with its >> purpose, any dissemination or disclosure, either whole or partial, is >> prohibited except formal approval. >> If you are not the addressee, you may not copy, forward, disclose or use >> any part of it. If you have received this message in error, please delete >> it and all copies from your system and notify the sender immediately by >> return message. >> E-mail communication cannot be guaranteed to be timely secure, error or >> virus-free. >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Jul 13 12:19:13 2023 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 13 Jul 2023 19:19:13 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: In any case, we need to activate PetscCheck in debug mode too. This could have been avoided. Il giorno gio 13 lug 2023 alle ore 18:23 Karin&NiKo ha scritto: > Thank you very much for the information Matt. > Unfortunately, I do not use DM.... :-( > > Le jeu. 13 juil. 2023 ? 13:44, Matthew Knepley a > ?crit : > >> On Thu, Jul 13, 2023 at 5:33?AM Pierre Jolivet >> wrote: >> >>> Dear Nicolas, >>> >>> On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas >>> wrote: >>> >>> Dear Pierre, >>> >>> You are absolutely right. I was using a --with-debugging=0 (aka release) >>> install and this is definitely an error. >>> Once I used my debug install, I found the way to fix my problem. The >>> solution is in the attached script: I first need to extract the correct >>> block from the PC operator's MatNest and then append the null space to it. >>> Anyway this is a bit tricky... >>> >>> >>> Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, you >>> name it. >>> You first need the initial PCSetUp() so that the bare minimum is put in >>> place, then you have to fetch things yourself and adapt it to your needs. >>> We had a similar discussion with the MEF++ people last week, there is >>> currently no way around this, AFAIK. >>> >> >> Actually, I hated this as well, so I built a way around it _if_ you are >> using a DM to define the problem. Then >> you can set a "nullspace constructor" to make it if the field you are >> talking about is ever extracted. You use DMSetNullSpaceConstructor(). I do >> this in SNES ex62 and ex69, and other examples. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Pierre >>> >>> Regards, >>> Nicolas >>> >>> ------------------------------ >>> *De :* pierre.jolivet at lip6.fr >>> *Envoy? :* mercredi 12 juillet 2023 19:52 >>> *? :* TARDIEU Nicolas >>> *Cc :* petsc-users at mcs.anl.gov >>> *Objet :* Re: [petsc-users] Near null space for a fieldsplit in petsc4py >>> >>> >>> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> > >>> > Dear PETSc team, >>> > >>> > In the attached example, I set up a block pc for a saddle-point >>> problem in petsc4py. The IS define the unknowns, namely some physical >>> quantity (phys) and a Lagrange multiplier (lags). >>> > I would like to attach a near null space to the physical block, in >>> order to get the best performance from an AMG pc. >>> > I have been trying hard, attaching it to the initial block, to the IS >>> but no matter what I am doing, when it comes to "ksp_view", no near null >>> space is attached to the matrix. >>> > >>> > Could you please help me figure out what I am doing wrong ? >>> >>> Are you using a double-precision 32-bit integers real build of PETSc? >>> Is it --with-debugging=0? >>> Because with my debug build, I get the following error (thus explaining >>> why it?s not attached to the KSP). >>> Traceback (most recent call last): >>> File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in >>> >>> ns = NullSpace().create(True, [v], comm=comm) >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> File "petsc4py/PETSc/Mat.pyx", line 5611, in >>> petsc4py.PETSc.NullSpace.create >>> petsc4py.PETSc.Error: error code 62 >>> [0] MatNullSpaceCreate() at >>> /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 >>> [0] Invalid argument >>> [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 >>> >>> Furthermore, if you set yourself the constant vector in the near >>> null-space, then the first argument of create() must be False, otherwise, >>> you?ll have twice the same vector, and you?ll end up with another error >>> (the vectors in the near null-space must be orthonormal). >>> If things still don?t work after those couple of fixes, please feel free >>> to send an up-to-date reproducer. >>> >>> Thanks, >>> Pierre >>> >>> > Thanks, >>> > Nicolas >>> > >>> > >>> > >>> > >>> > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >>> ?tablis ? l'intention exclusive des destinataires et les informations qui y >>> figurent sont strictement confidentielles. Toute utilisation de ce Message >>> non conforme ? sa destination, toute diffusion ou toute publication totale >>> ou partielle, est interdite sauf autorisation expresse. >>> > >>> > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit >>> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou >>> partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de >>> votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace >>> sur quelque support que ce soit. Nous vous remercions ?galement d'en >>> avertir imm?diatement l'exp?diteur par retour du message. >>> > >>> > Il est impossible de garantir que les communications par messagerie >>> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >>> erreur ou virus. >>> > ____________________________________________________ >>> > >>> > This message and any attachments (the 'Message') are intended solely >>> for the addressees. The information contained in this Message is >>> confidential. Any use of information contained in this Message not in >>> accord with its purpose, any dissemination or disclosure, either whole or >>> partial, is prohibited except formal approval. >>> > >>> > If you are not the addressee, you may not copy, forward, disclose or >>> use any part of it. If you have received this message in error, please >>> delete it and all copies from your system and notify the sender immediately >>> by return message. >>> > >>> > E-mail communication cannot be guaranteed to be timely secure, error >>> or virus-free. >>> > >>> >>> >>> Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >>> ?tablis ? l'intention exclusive des destinataires et les informations qui y >>> figurent sont strictement confidentielles. Toute utilisation de ce Message >>> non conforme ? sa destination, toute diffusion ou toute publication totale >>> ou partielle, est interdite sauf autorisation expresse. >>> Si vous n'?tes pas le destinataire de ce Message, il vous est interdit >>> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou >>> partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de >>> votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace >>> sur quelque support que ce soit. Nous vous remercions ?galement d'en >>> avertir imm?diatement l'exp?diteur par retour du message. >>> Il est impossible de garantir que les communications par messagerie >>> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >>> erreur ou virus. >>> ____________________________________________________ >>> This message and any attachments (the 'Message') are intended solely for >>> the addressees. The information contained in this Message is confidential. >>> Any use of information contained in this Message not in accord with its >>> purpose, any dissemination or disclosure, either whole or partial, is >>> prohibited except formal approval. >>> If you are not the addressee, you may not copy, forward, disclose or use >>> any part of it. If you have received this message in error, please delete >>> it and all copies from your system and notify the sender immediately by >>> return message. >>> E-mail communication cannot be guaranteed to be timely secure, error or >>> virus-free. >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Jul 13 12:19:38 2023 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 13 Jul 2023 19:19:38 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: clearly, I meant optimized mode Il giorno gio 13 lug 2023 alle ore 19:19 Stefano Zampini < stefano.zampini at gmail.com> ha scritto: > In any case, we need to activate PetscCheck in debug mode too. This could > have been avoided. > > Il giorno gio 13 lug 2023 alle ore 18:23 Karin&NiKo > ha scritto: > >> Thank you very much for the information Matt. >> Unfortunately, I do not use DM.... :-( >> >> Le jeu. 13 juil. 2023 ? 13:44, Matthew Knepley a >> ?crit : >> >>> On Thu, Jul 13, 2023 at 5:33?AM Pierre Jolivet >>> wrote: >>> >>>> Dear Nicolas, >>>> >>>> On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas >>>> wrote: >>>> >>>> Dear Pierre, >>>> >>>> You are absolutely right. I was using a --with-debugging=0 (aka >>>> release) install and this is definitely an error. >>>> Once I used my debug install, I found the way to fix my problem. The >>>> solution is in the attached script: I first need to extract the correct >>>> block from the PC operator's MatNest and then append the null space to it. >>>> Anyway this is a bit tricky... >>>> >>>> >>>> Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, you >>>> name it. >>>> You first need the initial PCSetUp() so that the bare minimum is put in >>>> place, then you have to fetch things yourself and adapt it to your needs. >>>> We had a similar discussion with the MEF++ people last week, there is >>>> currently no way around this, AFAIK. >>>> >>> >>> Actually, I hated this as well, so I built a way around it _if_ you are >>> using a DM to define the problem. Then >>> you can set a "nullspace constructor" to make it if the field you are >>> talking about is ever extracted. You use DMSetNullSpaceConstructor(). I do >>> this in SNES ex62 and ex69, and other examples. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Pierre >>>> >>>> Regards, >>>> Nicolas >>>> >>>> ------------------------------ >>>> *De :* pierre.jolivet at lip6.fr >>>> *Envoy? :* mercredi 12 juillet 2023 19:52 >>>> *? :* TARDIEU Nicolas >>>> *Cc :* petsc-users at mcs.anl.gov >>>> *Objet :* Re: [petsc-users] Near null space for a fieldsplit in >>>> petsc4py >>>> >>>> >>>> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users < >>>> petsc-users at mcs.anl.gov> wrote: >>>> > >>>> > Dear PETSc team, >>>> > >>>> > In the attached example, I set up a block pc for a saddle-point >>>> problem in petsc4py. The IS define the unknowns, namely some physical >>>> quantity (phys) and a Lagrange multiplier (lags). >>>> > I would like to attach a near null space to the physical block, in >>>> order to get the best performance from an AMG pc. >>>> > I have been trying hard, attaching it to the initial block, to the IS >>>> but no matter what I am doing, when it comes to "ksp_view", no near null >>>> space is attached to the matrix. >>>> > >>>> > Could you please help me figure out what I am doing wrong ? >>>> >>>> Are you using a double-precision 32-bit integers real build of PETSc? >>>> Is it --with-debugging=0? >>>> Because with my debug build, I get the following error (thus explaining >>>> why it?s not attached to the KSP). >>>> Traceback (most recent call last): >>>> File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in >>>> >>>> ns = NullSpace().create(True, [v], comm=comm) >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> File "petsc4py/PETSc/Mat.pyx", line 5611, in >>>> petsc4py.PETSc.NullSpace.create >>>> petsc4py.PETSc.Error: error code 62 >>>> [0] MatNullSpaceCreate() at >>>> /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 >>>> [0] Invalid argument >>>> [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 >>>> >>>> Furthermore, if you set yourself the constant vector in the near >>>> null-space, then the first argument of create() must be False, otherwise, >>>> you?ll have twice the same vector, and you?ll end up with another error >>>> (the vectors in the near null-space must be orthonormal). >>>> If things still don?t work after those couple of fixes, please feel >>>> free to send an up-to-date reproducer. >>>> >>>> Thanks, >>>> Pierre >>>> >>>> > Thanks, >>>> > Nicolas >>>> > >>>> > >>>> > >>>> > >>>> > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >>>> ?tablis ? l'intention exclusive des destinataires et les informations qui y >>>> figurent sont strictement confidentielles. Toute utilisation de ce Message >>>> non conforme ? sa destination, toute diffusion ou toute publication totale >>>> ou partielle, est interdite sauf autorisation expresse. >>>> > >>>> > Si vous n'?tes pas le destinataire de ce Message, il vous est >>>> interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser >>>> tout ou partie. Si vous avez re?u ce Message par erreur, merci de le >>>> supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder >>>> aucune trace sur quelque support que ce soit. Nous vous remercions >>>> ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. >>>> > >>>> > Il est impossible de garantir que les communications par messagerie >>>> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >>>> erreur ou virus. >>>> > ____________________________________________________ >>>> > >>>> > This message and any attachments (the 'Message') are intended solely >>>> for the addressees. The information contained in this Message is >>>> confidential. Any use of information contained in this Message not in >>>> accord with its purpose, any dissemination or disclosure, either whole or >>>> partial, is prohibited except formal approval. >>>> > >>>> > If you are not the addressee, you may not copy, forward, disclose or >>>> use any part of it. If you have received this message in error, please >>>> delete it and all copies from your system and notify the sender immediately >>>> by return message. >>>> > >>>> > E-mail communication cannot be guaranteed to be timely secure, error >>>> or virus-free. >>>> > >>>> >>>> >>>> Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >>>> ?tablis ? l'intention exclusive des destinataires et les informations qui y >>>> figurent sont strictement confidentielles. Toute utilisation de ce Message >>>> non conforme ? sa destination, toute diffusion ou toute publication totale >>>> ou partielle, est interdite sauf autorisation expresse. >>>> Si vous n'?tes pas le destinataire de ce Message, il vous est interdit >>>> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou >>>> partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de >>>> votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace >>>> sur quelque support que ce soit. Nous vous remercions ?galement d'en >>>> avertir imm?diatement l'exp?diteur par retour du message. >>>> Il est impossible de garantir que les communications par messagerie >>>> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute >>>> erreur ou virus. >>>> ____________________________________________________ >>>> This message and any attachments (the 'Message') are intended solely >>>> for the addressees. The information contained in this Message is >>>> confidential. Any use of information contained in this Message not in >>>> accord with its purpose, any dissemination or disclosure, either whole or >>>> partial, is prohibited except formal approval. >>>> If you are not the addressee, you may not copy, forward, disclose or >>>> use any part of it. If you have received this message in error, please >>>> delete it and all copies from your system and notify the sender immediately >>>> by return message. >>>> E-mail communication cannot be guaranteed to be timely secure, error or >>>> virus-free. >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > Stefano > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Chamberland at giref.ulaval.ca Thu Jul 13 16:30:02 2023 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Thu, 13 Jul 2023 23:30:02 +0200 Subject: [petsc-users] Near null space for a fieldsplit in petsc4py In-Reply-To: References: <456C7E69-1776-43E5-ABA0-E3D0E7A87CCA@lip6.fr> Message-ID: Hi everyone, I'm joining this discussion because we're testing a simple way to simplify the current object naming and configuration system in PETSc. The solution we're testing uses the same philosophy as the option database keys: a "default" database. We're creating a "default" PetscObjectList that gives access to a global list of pairs: . With this, we can simply "name" and add all our PetscObjects to it. At run time, we create an option that holds the name of the PetscObject to be passed as a parameter to whatever we want to configure. For example, let's say you want to pass an optional matrix named "preSchur" to a specific PC. It could be configured like this: Mat MyMat; PetscObjectSetName(MyMat, "preSchur"); PetscObjectListAdd(gGlobalList, "preSchur", (PetscObject)MyMat); /* Here MyMat is made available into the global object list with the key "preSchur" */ Then, in a PC, you can retrieve the matrix by using both the options database keys and the objects database. For example, if a PC looks for options like: -pc_fieldsplit_schur_precondition user -pc_fieldsplit_schur_precondition_user_mat preSchur It just needs to look if the key "pc_fieldsplit_schur_precondition_user_mat" has been set, and if so, retrieve the value, which is the name of the matrix to be retrieved from the global list. With this approach, all PCs, KSPs, and other objects can be given objects at setup time (or later) with proper naming of the options and objects. Some key features of this approach include: - If a user misspells an option, it will be visible in the unused options. - Any new options to any PC or KSP do not need a new function to use them; just a new option key is needed. - All software that uses PETSc as a library can have a very "light" interface to it: just name all objects and let users "configure" them! - This is almost the same idea as "composing objects," which is already used in many places in PETSc. It's simpler to use and really relaxes the constraints on "when" you can configure a PC. I think this is a promising approach to simplifying the object naming and configuration system in PETSc. I'm interested in hearing your feedback. Thanks, Eric On 2023-07-13 13:44, Matthew Knepley wrote: > On Thu, Jul 13, 2023 at 5:33?AM Pierre Jolivet > wrote: > > Dear Nicolas, > >> On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas >> wrote: >> >> Dear Pierre, >> >> You are absolutely right. I was using a --with-debugging=0 (aka >> release) install and this is definitely an error. >> Once I used my debug install, I found the way to fix my problem. >> The solution is in the attached script: I first need to extract >> the correct block from the PC operator's MatNest and then append >> the null space to it. >> Anyway this is a bit tricky... > > Yep, it?s the same with all ?nested? solvers, fieldsplit, ASM, MG, > you name it. > You first need the initial PCSetUp() so that the bare minimum is > put in place, then you have to fetch things yourself and adapt it > to your needs. > We had a similar discussion with the MEF++ people last week, there > is currently no way around this, AFAIK. > > > Actually, I hated this as well, so I built a way around it _if_ you > are using a DM to define the problem. Then > you can set a "nullspace constructor" to make it if the field you are > talking about is ever extracted. You use?DMSetNullSpaceConstructor(). > I do this in SNES ex62 and ex69, and other examples. > > ? Thanks, > > ? ? ?Matt > > Thanks, > Pierre > >> Regards, >> Nicolas >> >> ------------------------------------------------------------------------ >> *De :*pierre.jolivet at lip6.fr >> *Envoy? :*mercredi 12 juillet 2023 19:52 >> *? :*TARDIEU Nicolas >> *Cc?:*petsc-users at mcs.anl.gov >> *Objet :*Re: [petsc-users] Near null space for a fieldsplit in >> petsc4py >> >> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users >> wrote: >> > >> > Dear PETSc team, >> > >> > In the attached example, I set up a block pc for a saddle-point >> problem in petsc4py. The IS define the unknowns, namely some >> physical quantity (phys) and a Lagrange multiplier (lags). >> > I would like to attach a near null space to the physical block, >> in order to get the best performance from an AMG pc. >> > I have been trying hard, attaching it to the initial block, to >> the IS but no matter what I am doing, when it comes to >> "ksp_view", no near null space is attached to the matrix. >> > >> > Could you please help me figure out what I am doing wrong ? >> >> Are you using a double-precision 32-bit integers real build of PETSc? >> Is it --with-debugging=0? >> Because with my debug build, I get the following error (thus >> explaining why it?s not attached to the KSP). >> Traceback (most recent call last): >> ? File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, >> in >> ??? ns = NullSpace().create(True, [v], comm=comm) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> ? File "petsc4py/PETSc/Mat.pyx", line 5611, in >> petsc4py.PETSc.NullSpace.create >> petsc4py.PETSc.Error: error code 62 >> [0] MatNullSpaceCreate() at >> /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249 >> [0] Invalid argument >> [0] Vector 0 must have 2-norm of 1.0, it is 22.3159 >> >> Furthermore, if you set yourself the constant vector in the near >> null-space, then the first argument of create() must be False, >> otherwise, you?ll have twice the same vector, and you?ll end up >> with another error (the vectors in the near null-space must be >> orthonormal). >> If things still don?t work after those couple of fixes, please >> feel free to send an up-to-date reproducer. >> >> Thanks, >> Pierre >> >> > Thanks, >> > Nicolas >> > >> > >> > >> > >> > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') >> sont ?tablis ? l'intention exclusive des destinataires et les >> informations qui y figurent sont strictement confidentielles. >> Toute utilisation de ce Message non conforme ? sa destination, >> toute diffusion ou toute publication totale ou partielle, est >> interdite sauf autorisation expresse. >> > >> > Si vous n'?tes pas le destinataire de ce Message, il vous est >> interdit de le copier, de le faire suivre, de le divulguer ou >> d'en utiliser tout ou partie. Si vous avez re?u ce Message par >> erreur, merci de le supprimer de votre syst?me, ainsi que toutes >> ses copies, et de n'en garder aucune trace sur quelque support >> que ce soit. Nous vous remercions ?galement d'en avertir >> imm?diatement l'exp?diteur par retour du message. >> > >> > Il est impossible de garantir que les communications par >> messagerie ?lectronique arrivent en temps utile, sont s?curis?es >> ou d?nu?es de toute erreur ou virus. >> > ____________________________________________________ >> > >> > This message and any attachments (the 'Message') are intended >> solely for the addressees. The information contained in this >> Message is confidential. Any use of information contained in this >> Message not in accord with its purpose, any dissemination or >> disclosure, either whole or partial, is prohibited except formal >> approval. >> > >> > If you are not the addressee, you may not copy, forward, >> disclose or use any part of it. If you have received this message >> in error, please delete it and all copies from your system and >> notify the sender immediately by return message. >> > >> > E-mail communication cannot be guaranteed to be timely secure, >> error or virus-free. >> > >> >> >> Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') >> sont ?tablis ? l'intention exclusive des destinataires et les >> informations qui y figurent sont strictement confidentielles. >> Toute utilisation de ce Message non conforme ? sa destination, >> toute diffusion ou toute publication totale ou partielle, est >> interdite sauf autorisation expresse. >> Si vous n'?tes pas le destinataire de ce Message, il vous est >> interdit de le copier, de le faire suivre, de le divulguer ou >> d'en utiliser tout ou partie. Si vous avez re?u ce Message par >> erreur, merci de le supprimer de votre syst?me, ainsi que toutes >> ses copies, et de n'en garder aucune trace sur quelque support >> que ce soit. Nous vous remercions ?galement d'en avertir >> imm?diatement l'exp?diteur par retour du message. >> Il est impossible de garantir que les communications par >> messagerie ?lectronique arrivent en temps utile, sont s?curis?es >> ou d?nu?es de toute erreur ou virus. >> ____________________________________________________ >> This message and any attachments (the 'Message') are intended >> solely for the addressees. The information contained in this >> Message is confidential. Any use of information contained in this >> Message not in accord with its purpose, any dissemination or >> disclosure, either whole or partial, is prohibited except formal >> approval. >> If you are not the addressee, you may not copy, forward, disclose >> or use any part of it. If you have received this message in >> error, please delete it and all copies from your system and >> notify the sender immediately by return message. >> E-mail communication cannot be guaranteed to be timely secure, >> error or virus-free. >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Universit? Laval (418) 656-2131 poste 41 22 42 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cho at slac.stanford.edu Fri Jul 14 18:56:43 2023 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Fri, 14 Jul 2023 23:56:43 +0000 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: Message-ID: I managed to pass the following options to PETSc using a GPU node on Perlmutter. -mat_type aijcusparse -vec_type cuda -log_view -options_left Below is a summary of the test using 4 MPI tasks and 1 GPU per task. o #PETSc Option Table entries: ???-log_view ???-mat_type aijcusparse -options_left -vec_type cuda #End of PETSc Option Table entries WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-mat_type value: aijcusparse The -mat_type option has not been used. In the application code, we use ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); o The percent flops on the GPU for KSPSolve is 17%. In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance? Thanks, Cho ________________________________ From: Ng, Cho-Kuen Sent: Friday, June 30, 2023 7:57 AM To: Barry Smith ; Mark Adams Cc: Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using PETSc GPU backend Barry, Mark and Matt, Thank you all for the suggestions. I will modify the code so we can pass runtime options. Cho ________________________________ From: Barry Smith Sent: Friday, June 30, 2023 7:01 AM To: Mark Adams Cc: Matthew Knepley ; Ng, Cho-Kuen ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using PETSc GPU backend Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know. On Jun 30, 2023, at 9:30 AM, Mark Adams wrote: PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run: a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left Mark On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley > wrote: On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users > wrote: Mark, The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right? PETSc will read command line argument automatically in PetscInitialize() unless you shut it off. Thanks, Matt Cho ________________________________ From: Ng, Cho-Kuen > Sent: Thursday, June 29, 2023 8:32 PM To: Mark Adams > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Mark, Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options? Cho ________________________________ From: Mark Adams > Sent: Thursday, June 29, 2023 5:55 PM To: Ng, Cho-Kuen > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0. The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left. Mark On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users > wrote: I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? Thanks, Cho -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 14 19:57:16 2023 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 14 Jul 2023 20:57:16 -0400 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: Message-ID: On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen wrote: > I managed to pass the following options to PETSc using a GPU node on > Perlmutter. > > -mat_type aijcusparse -vec_type cuda -log_view -options_left > > Below is a summary of the test using 4 MPI tasks and 1 GPU per task. > > o #PETSc Option Table entries: > -log_view > -mat_type aijcusparse > -options_left > -vec_type cuda > #End of PETSc Option Table entries > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > There is one unused database option. It is: > Option left: name:-mat_type value: aijcusparse > > The -mat_type option has not been used. In the application code, we use > > ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, > d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); > > If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line. Thanks, Matt > o The percent flops on the GPU for KSPSolve is 17%. > > In comparison with a CPU run using 16 MPI tasks, the GPU run is an order > of magnitude slower. How can I improve the GPU performance? > > Thanks, > Cho > ------------------------------ > *From:* Ng, Cho-Kuen > *Sent:* Friday, June 30, 2023 7:57 AM > *To:* Barry Smith ; Mark Adams > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Barry, Mark and Matt, > > Thank you all for the suggestions. I will modify the code so we can pass > runtime options. > > Cho > ------------------------------ > *From:* Barry Smith > *Sent:* Friday, June 30, 2023 7:01 AM > *To:* Mark Adams > *Cc:* Matthew Knepley ; Ng, Cho-Kuen < > cho at slac.stanford.edu>; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > > Note that options like -mat_type aijcusparse -vec_type cuda only work > if the program is set up to allow runtime swapping of matrix and vector > types. If you have a call to MatCreateMPIAIJ() or other specific types then > then these options do nothing but because Mark had you use -options_left > the program will tell you at the end that it did not use the option so you > will know. > > On Jun 30, 2023, at 9:30 AM, Mark Adams wrote: > > PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args > and you run: > > a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left > > Mark > > On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley wrote: > > On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Mark, > > The application code reads in parameters from an input file, where we can > put the PETSc runtime options. Then we pass the options to > PetscInitialize(...). Does that sounds right? > > > PETSc will read command line argument automatically in PetscInitialize() > unless you shut it off. > > Thanks, > > Matt > > > Cho > ------------------------------ > *From:* Ng, Cho-Kuen > *Sent:* Thursday, June 29, 2023 8:32 PM > *To:* Mark Adams > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Mark, > > Thanks for the information. How do I put the runtime options for the > executable, say, a.out, which does not have the provision to append > arguments? Do I need to change the C++ main to read in the options? > > Cho > ------------------------------ > *From:* Mark Adams > *Sent:* Thursday, June 29, 2023 5:55 PM > *To:* Ng, Cho-Kuen > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Run with options: -mat_type aijcusparse -vec_type cuda -log_view > -options_left > > The last column of the performance data (from -log_view) will be the > percent flops on the GPU. Check that that is > 0. > > The end of the output will list the options that were used and options > that were _not_ used (if any). Check that there are no options left. > > Mark > > On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and > used it by "spack load petsc/fwge6pf". Then I compiled the application > code (purely CPU code) linking to the petsc package, hoping that I can get > performance improvement using the petsc GPU backend. However, the timing > was the same using the same number of MPI tasks with and without GPU > accelerators. Have I missed something in the process, for example, setting > up PETSc options at runtime to use the GPU backend? > > Thanks, > Cho > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cho at slac.stanford.edu Sat Jul 15 00:44:27 2023 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Sat, 15 Jul 2023 05:44:27 +0000 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: Message-ID: Matt, After inserting 2 lines in the code: ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); "There are no unused options." However, there is no improvement on the GPU performance. Thanks, Cho ________________________________ From: Matthew Knepley Sent: Friday, July 14, 2023 5:57 PM To: Ng, Cho-Kuen Cc: Barry Smith ; Mark Adams ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using PETSc GPU backend On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen > wrote: I managed to pass the following options to PETSc using a GPU node on Perlmutter. -mat_type aijcusparse -vec_type cuda -log_view -options_left Below is a summary of the test using 4 MPI tasks and 1 GPU per task. o #PETSc Option Table entries: ???-log_view ???-mat_type aijcusparse -options_left -vec_type cuda #End of PETSc Option Table entries WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-mat_type value: aijcusparse The -mat_type option has not been used. In the application code, we use ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line. Thanks, Matt o The percent flops on the GPU for KSPSolve is 17%. In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance? Thanks, Cho ________________________________ From: Ng, Cho-Kuen > Sent: Friday, June 30, 2023 7:57 AM To: Barry Smith >; Mark Adams > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Barry, Mark and Matt, Thank you all for the suggestions. I will modify the code so we can pass runtime options. Cho ________________________________ From: Barry Smith > Sent: Friday, June 30, 2023 7:01 AM To: Mark Adams > Cc: Matthew Knepley >; Ng, Cho-Kuen >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know. On Jun 30, 2023, at 9:30 AM, Mark Adams > wrote: PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run: a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left Mark On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley > wrote: On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users > wrote: Mark, The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right? PETSc will read command line argument automatically in PetscInitialize() unless you shut it off. Thanks, Matt Cho ________________________________ From: Ng, Cho-Kuen > Sent: Thursday, June 29, 2023 8:32 PM To: Mark Adams > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Mark, Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options? Cho ________________________________ From: Mark Adams > Sent: Thursday, June 29, 2023 5:55 PM To: Ng, Cho-Kuen > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0. The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left. Mark On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users > wrote: I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? Thanks, Cho -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jul 15 07:03:28 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 15 Jul 2023 08:03:28 -0400 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: Message-ID: On Sat, Jul 15, 2023 at 1:44?AM Ng, Cho-Kuen wrote: > Matt, > > After inserting 2 lines in the code: > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > > ierr = MatSetFromOptions(A);CHKERRQ(ierr); > ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, > d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); > > "There are no unused options." However, there is no improvement on the GPU > performance. > 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual. 2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions(). THanks, Matt > Thanks, > Cho > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Friday, July 14, 2023 5:57 PM > *To:* Ng, Cho-Kuen > *Cc:* Barry Smith ; Mark Adams ; > petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen > wrote: > > I managed to pass the following options to PETSc using a GPU node on > Perlmutter. > > -mat_type aijcusparse -vec_type cuda -log_view -options_left > > Below is a summary of the test using 4 MPI tasks and 1 GPU per task. > > o #PETSc Option Table entries: > -log_view > -mat_type aijcusparse > -options_left > -vec_type cuda > #End of PETSc Option Table entries > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > There is one unused database option. It is: > Option left: name:-mat_type value: aijcusparse > > The -mat_type option has not been used. In the application code, we use > > ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, > d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); > > > If you create the Mat this way, then you need MatSetFromOptions() in order > to set the type from the command line. > > Thanks, > > Matt > > > o The percent flops on the GPU for KSPSolve is 17%. > > In comparison with a CPU run using 16 MPI tasks, the GPU run is an order > of magnitude slower. How can I improve the GPU performance? > > Thanks, > Cho > ------------------------------ > *From:* Ng, Cho-Kuen > *Sent:* Friday, June 30, 2023 7:57 AM > *To:* Barry Smith ; Mark Adams > *Cc:* Matthew Knepley ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Barry, Mark and Matt, > > Thank you all for the suggestions. I will modify the code so we can pass > runtime options. > > Cho > ------------------------------ > *From:* Barry Smith > *Sent:* Friday, June 30, 2023 7:01 AM > *To:* Mark Adams > *Cc:* Matthew Knepley ; Ng, Cho-Kuen < > cho at slac.stanford.edu>; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > > Note that options like -mat_type aijcusparse -vec_type cuda only work > if the program is set up to allow runtime swapping of matrix and vector > types. If you have a call to MatCreateMPIAIJ() or other specific types then > then these options do nothing but because Mark had you use -options_left > the program will tell you at the end that it did not use the option so you > will know. > > On Jun 30, 2023, at 9:30 AM, Mark Adams wrote: > > PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args > and you run: > > a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left > > Mark > > On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley wrote: > > On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Mark, > > The application code reads in parameters from an input file, where we can > put the PETSc runtime options. Then we pass the options to > PetscInitialize(...). Does that sounds right? > > > PETSc will read command line argument automatically in PetscInitialize() > unless you shut it off. > > Thanks, > > Matt > > > Cho > ------------------------------ > *From:* Ng, Cho-Kuen > *Sent:* Thursday, June 29, 2023 8:32 PM > *To:* Mark Adams > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Mark, > > Thanks for the information. How do I put the runtime options for the > executable, say, a.out, which does not have the provision to append > arguments? Do I need to change the C++ main to read in the options? > > Cho > ------------------------------ > *From:* Mark Adams > *Sent:* Thursday, June 29, 2023 5:55 PM > *To:* Ng, Cho-Kuen > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Using PETSc GPU backend > > Run with options: -mat_type aijcusparse -vec_type cuda -log_view > -options_left > > The last column of the performance data (from -log_view) will be the > percent flops on the GPU. Check that that is > 0. > > The end of the output will list the options that were used and options > that were _not_ used (if any). Check that there are no options left. > > Mark > > On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and > used it by "spack load petsc/fwge6pf". Then I compiled the application > code (purely CPU code) linking to the petsc package, hoping that I can get > performance improvement using the petsc GPU backend. However, the timing > was the same using the same number of MPI tasks with and without GPU > accelerators. Have I missed something in the process, for example, setting > up PETSc options at runtime to use the GPU backend? > > Thanks, > Cho > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jul 15 10:36:08 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 15 Jul 2023 11:36:08 -0400 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: Message-ID: <10FFD366-3B5A-4B3D-A5AF-8BA0C093C882@petsc.dev> Cho, We currently have a crappy API for turning on GPU support, and our documentation is misleading in places. People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect. This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda If you use DM to create the matrices and vectors then you can use -dm_mat_type aijcusparse -dm_vec_type cuda Sorry for the confusion. Barry > On Jul 15, 2023, at 8:03 AM, Matthew Knepley wrote: > > On Sat, Jul 15, 2023 at 1:44?AM Ng, Cho-Kuen > wrote: >> Matt, >> >> After inserting 2 lines in the code: >> >> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> ierr = MatSetFromOptions(A);CHKERRQ(ierr); >> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >> "There are no unused options." However, there is no improvement on the GPU performance. > > 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual. > > 2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions(). > > THanks, > > Matt > >> Thanks, >> Cho >> >> From: Matthew Knepley > >> Sent: Friday, July 14, 2023 5:57 PM >> To: Ng, Cho-Kuen > >> Cc: Barry Smith >; Mark Adams >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen > wrote: >> I managed to pass the following options to PETSc using a GPU node on Perlmutter. >> >> -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >> Below is a summary of the test using 4 MPI tasks and 1 GPU per task. >> >> o #PETSc Option Table entries: >> ???-log_view >> ???-mat_type aijcusparse >> -options_left >> -vec_type cuda >> #End of PETSc Option Table entries >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-mat_type value: aijcusparse >> >> The -mat_type option has not been used. In the application code, we use >> >> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >> >> If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line. >> >> Thanks, >> >> Matt >> >> o The percent flops on the GPU for KSPSolve is 17%. >> >> In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance? >> >> Thanks, >> Cho >> From: Ng, Cho-Kuen > >> Sent: Friday, June 30, 2023 7:57 AM >> To: Barry Smith >; Mark Adams > >> Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> Barry, Mark and Matt, >> >> Thank you all for the suggestions. I will modify the code so we can pass runtime options. >> >> Cho >> From: Barry Smith > >> Sent: Friday, June 30, 2023 7:01 AM >> To: Mark Adams > >> Cc: Matthew Knepley >; Ng, Cho-Kuen >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> >> Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know. >> >>> On Jun 30, 2023, at 9:30 AM, Mark Adams > wrote: >>> >>> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run: >>> >>> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left >>> >>> Mark >>> >>> On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley > wrote: >>> On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users > wrote: >>> Mark, >>> >>> The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right? >>> >>> PETSc will read command line argument automatically in PetscInitialize() unless you shut it off. >>> >>> Thanks, >>> >>> Matt >>> >>> Cho >>> From: Ng, Cho-Kuen > >>> Sent: Thursday, June 29, 2023 8:32 PM >>> To: Mark Adams > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Mark, >>> >>> Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options? >>> >>> Cho >>> From: Mark Adams > >>> Sent: Thursday, June 29, 2023 5:55 PM >>> To: Ng, Cho-Kuen > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left >>> >>> The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0. >>> >>> The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left. >>> >>> Mark >>> >>> On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users > wrote: >>> I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? >>> >>> Thanks, >>> Cho >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rochan.upadhyay at gmail.com Sat Jul 15 18:08:51 2023 From: rochan.upadhyay at gmail.com (Rochan Upadhyay) Date: Sat, 15 Jul 2023 18:08:51 -0500 Subject: [petsc-users] Problems in use of VecNest in Petsc? Message-ID: Dear Petsc developers, Is the use of VecNest format supported for field-split type preconditioner or for use with MatNest formats ? A quick change to example snes/tutorials/ex70.c seems to suggest that it is not possible. For example the changes : Vec subx[2], subb[2]; (declared in Stokes struct) In routine StokesSetupVectors(Stokes *s) : MatCreateVecs(s->subA[0], &s->subx[0], &s->subb[0]); MatCreateVecs(s->subA[3], &s->subx[1], &s->subb[1]); // IS is_rows[2]; MatNestGetISs(s->A, is_rows, NULL); // VecCreateNest(PETSC_COMM_WORLD, 2, is_rows, s->subx, &s->x); VecCreateNest(PETSC_COMM_WORLD, 2, is_rows, s->subb, &s->b); VecAssemblyBegin(s->x); VecAssemblyEnd(s->x); VecAssemblyBegin(s->b); VecAssemblyEnd(s->b); .... The serial case runs fine but the parallel with np = 2 fails with the error message copied below. Could you find out what I am doing wrong ? In th error message, which vector(s) needs assembling ? I would normally not use VecNest but many finite element packages that allow hooking up with petsc seem to have their block vectors of Petsc VecType VecNest Any help/insights would be appreciated. Regards, Rochan [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, Jun 29, 2023 [0]PETSC ERROR: ./ex70 on a named localhost.localdomain by user Sat Jul 15 17:51:01 2023 [0]PETSC ERROR: Configure options --with-clean=yes --with-debugging=no --with-mpi=1 --with-mpi-dir=/home/user/software_installs/mpich-4.0.2/ --prefix=/home/user/software_installs/petsc-3.19.3/ --PETSC_ARCH=linux-gnu-opt --PETSC_DIR=/home/user/software_downloads/petsc-3.19.3/ --with-cxx-dialect=C++11 --with-hypre=1 --download-hypre=yes --with-suitesparse=1 --download-suitesparse=yes --with-metis=1 --download-metis=yes --with-shared-libraries=1 --with-zlib=1 --download-zlib=no --with-superlu_dist=1 --download-superlu_dist=yes --with-mumps=1 --download-mumps=yes --with-parmetis=1 --download-parmetis=yes --with-hdf5=yes --download-hdf5=1 --with-chaco=1 --download-chaco=yes --with-scalapack=1 --download-scalapack=yes --with-p4est=1 --download-p4est=yes --download-fblaslapack=1 --with-strumpack=1 --download-strumpack=yes [0]PETSC ERROR: #1 VecCopy() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/interface/vector.c:1684 [0]PETSC ERROR: #2 VecCopy_Nest() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/impls/nest/vecnest.c:58 [0]PETSC ERROR: #3 VecCopy() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/interface/vector.c:1723 [0]PETSC ERROR: #4 KSPInitialResidual() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itres.c:60 [0]PETSC ERROR: #5 KSPSolve_GMRES() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/impls/gmres/gmres.c:226 [0]PETSC ERROR: #6 KSPSolve_Private() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itfunc.c:898 [0]PETSC ERROR: #7 KSPSolve() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itfunc.c:1070 [0]PETSC ERROR: #8 main() at /home/user/Projects/Misc/Petsc/SNES/ex70.c:684 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_monitor (source: command line) [0]PETSC ERROR: -nx 4 (source: command line) [0]PETSC ERROR: -ny 8 (source: command line) [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- Abort(73) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 73) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sat Jul 15 22:31:14 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 15 Jul 2023 23:31:14 -0400 Subject: [petsc-users] Problems in use of VecNest in Petsc? In-Reply-To: References: Message-ID: <98CF3D19-E2E7-44A4-981A-2938C7650A00@petsc.dev> It is in the loop below that it errors due to an unassembled vector. T static PetscErrorCode VecCopy_Nest(Vec x, Vec y) { Vec_Nest *bx = (Vec_Nest *)x->data; Vec_Nest *by = (Vec_Nest *)y->data; PetscInt i; PetscFunctionBegin; VecNestCheckCompatible2(x, 1, y, 2); for (i = 0; i < bx->nb; i++) PetscCall(VecCopy(bx->v[i], by->v[i])); PetscFunctionReturn(PETSC_SUCCESS); } thus it is one of the individual vectors that make up the full vectors that is marked as not assembled In ex70.c I see PetscErrorCode StokesExactSolution(Stokes *s) { PetscInt row, start, end, i, j; PetscScalar val; Vec y0, y1; PetscFunctionBeginUser; /* velocity part */ PetscCall(VecGetSubVector(s->y, s->isg[0], &y0)); PetscCall(VecGetOwnershipRange(y0, &start, &end)); for (row = start; row < end; row++) { PetscCall(StokesGetPosition(s, row, &i, &j)); if (row < s->nx * s->ny) { val = StokesExactVelocityX(j * s->hy + s->hy / 2); } else { val = 0; } PetscCall(VecSetValue(y0, row, val, INSERT_VALUES)); } PetscCall(VecRestoreSubVector(s->y, s->isg[0], &y0)); VecSetValues() is called on y0 but VecAssemblyBegin/End is never called hence that vector is not assembled and will result in the error reported. All the vectors in this example on which VecSetValue[s] is called need to have VecAssemblyBegin/End. Barry For sequential vectors where entries set with VecSetValues() are inserted directly in the array it is possible to "get away" with not calling VecAssemblyBegin/End but on should never do that so even in the sequential case this example is technically buggy > On Jul 15, 2023, at 7:08 PM, Rochan Upadhyay wrote: > > Dear Petsc developers, > Is the use of VecNest format supported for field-split type preconditioner or for use with MatNest formats ? > A quick change to example snes/tutorials/ex70.c seems to suggest that it is not possible. > > For example the changes : > Vec subx[2], subb[2]; (declared in Stokes struct) > In routine StokesSetupVectors(Stokes *s) : > MatCreateVecs(s->subA[0], &s->subx[0], &s->subb[0]); > MatCreateVecs(s->subA[3], &s->subx[1], &s->subb[1]); > // > IS is_rows[2]; > MatNestGetISs(s->A, is_rows, NULL); > // > VecCreateNest(PETSC_COMM_WORLD, 2, is_rows, s->subx, &s->x); > VecCreateNest(PETSC_COMM_WORLD, 2, is_rows, s->subb, &s->b); > VecAssemblyBegin(s->x); > VecAssemblyEnd(s->x); > VecAssemblyBegin(s->b); > VecAssemblyEnd(s->b); > .... > The serial case runs fine but the parallel with np = 2 fails with the error message copied below. Could you find out what I am doing wrong ? In th error message, which vector(s) needs assembling ? > I would normally not use VecNest but many finite element packages that allow hooking up with petsc seem to have their block vectors of Petsc VecType VecNest > Any help/insights would be appreciated. > Regards, > Rochan > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.19.3, Jun 29, 2023 > [0]PETSC ERROR: ./ex70 on a named localhost.localdomain by user Sat Jul 15 17:51:01 2023 > [0]PETSC ERROR: Configure options --with-clean=yes --with-debugging=no --with-mpi=1 --with-mpi-dir=/home/user/software_installs/mpich-4.0.2/ --prefix=/home/user/software_installs/petsc-3.19.3/ --PETSC_ARCH=linux-gnu-opt --PETSC_DIR=/home/user/software_downloads/petsc-3.19.3/ --with-cxx-dialect=C++11 --with-hypre=1 --download-hypre=yes --with-suitesparse=1 --download-suitesparse=yes --with-metis=1 --download-metis=yes --with-shared-libraries=1 --with-zlib=1 --download-zlib=no --with-superlu_dist=1 --download-superlu_dist=yes --with-mumps=1 --download-mumps=yes --with-parmetis=1 --download-parmetis=yes --with-hdf5=yes --download-hdf5=1 --with-chaco=1 --download-chaco=yes --with-scalapack=1 --download-scalapack=yes --with-p4est=1 --download-p4est=yes --download-fblaslapack=1 --with-strumpack=1 --download-strumpack=yes > [0]PETSC ERROR: #1 VecCopy() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/interface/vector.c:1684 > [0]PETSC ERROR: #2 VecCopy_Nest() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/impls/nest/vecnest.c:58 > [0]PETSC ERROR: #3 VecCopy() at /home/user/software_downloads/petsc-3.19.3/src/vec/vec/interface/vector.c:1723 > [0]PETSC ERROR: #4 KSPInitialResidual() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itres.c:60 > [0]PETSC ERROR: #5 KSPSolve_GMRES() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/impls/gmres/gmres.c:226 > [0]PETSC ERROR: #6 KSPSolve_Private() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itfunc.c:898 > [0]PETSC ERROR: #7 KSPSolve() at /home/user/software_downloads/petsc-3.19.3/src/ksp/ksp/interface/itfunc.c:1070 > [0]PETSC ERROR: #8 main() at /home/user/Projects/Misc/Petsc/SNES/ex70.c:684 > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -ksp_monitor (source: command line) > [0]PETSC ERROR: -nx 4 (source: command line) > [0]PETSC ERROR: -ny 8 (source: command line) > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > Abort(73) on node 0 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 73) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrean01 at gmail.com Sun Jul 16 10:45:22 2023 From: jcrean01 at gmail.com (Jared Crean) Date: Sun, 16 Jul 2023 09:45:22 -0600 Subject: [petsc-users] Bug in log_view printing Message-ID: <53873868-8b32-e5b2-c826-102ab448174e@gmail.com> Hello, ? I stumbled upon a small bug in the printing of the data when running Petsc with -log_view.? The entire table is attached, but here is an example row: ------------------------------------------------------------------------------------------------------------------------ Event??????????????? Count????? Time (sec) Flop????????????????????????????? --- Global ---? --- Stage ---- Total ?????????????????? Max Ratio? Max???? Ratio?? Max? Ratio? Mess AvgLen? Reduct? %T %F %M %L %R? %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided????????? 7 1.0 1.3282e-0271.6 0.00e+00 0.0 6.0e+01 4.0e+00 5.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0???? 0 ? Notice how the Time Max and Time Ratio values are missing a space separator (the other rows have only two digits for the exponents, I think there should be a space after the e-02 and before the 7). ? Configuration details: ??? * Petsc 3.17 ??? * Arm64 architecture (and AWS c7g.xlarge instance, which use the new Graviton3 processor) ??? * ubuntu 22.04 ??? * MPICH MPI (version 4.0, I think, whatever apt-get install mpich gives) ??? * clang-15 compiler ? Jared Crean -------------- next part -------------- ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 7 1.0 1.3282e-0271.6 0.00e+00 0.0 6.0e+01 4.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecTDot 6852 1.0 4.5610e-01 4.2 1.70e+08 1.1 0.0e+00 0.0e+00 6.9e+03 1 2 0 0 30 1 2 0 0 30 1434 VecNorm 3721 1.0 2.8767e+00143.9 9.22e+07 1.1 0.0e+00 0.0e+00 3.7e+03 8 1 0 0 16 8 1 0 0 16 123 VecCopy 590 1.0 3.3962e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 11458 1.0 5.3920e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 6852 1.0 3.8920e-02 1.1 1.70e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 16807 VecAYPX 3131 1.0 2.4083e-02 1.4 7.75e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 12411 VecScatterBegin 18605 1.0 2.1698e-01 1.1 0.00e+00 0.0 8.9e+04 2.6e+04 3.0e+00 1 0 94 43 0 1 0 94 43 0 0 VecScatterEnd 18605 1.0 5.6709e+00 9.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 SFSetGraph 7 1.0 1.3217e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 7 1.0 1.3869e-02 6.2 0.00e+00 0.0 1.2e+02 5.6e+03 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceBegin 3721 1.0 1.9801e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceEnd 3721 1.0 1.6752e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFPack 18605 1.0 8.0476e-02 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFUnpack 18605 1.0 5.8449e-03 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3426 1.0 9.8399e-01 1.2 2.33e+09 1.1 4.1e+04 2.7e+04 1.0e+00 3 23 43 21 0 3 23 43 21 0 8914 MatSolve 3721 1.0 4.1004e+00 3.3 9.53e+09 3.7 0.0e+00 0.0e+00 0.0e+00 8 44 0 0 0 8 44 0 0 0 4227 MatLUFactorNum 295 1.0 3.6820e+00 3.7 5.19e+09 3.9 0.0e+00 0.0e+00 0.0e+00 6 24 0 0 0 6 24 0 0 0 2528 MatILUFactorSym 1 1.0 4.1936e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 2 1.0 2.7529e-03 1.2 0.00e+00 0.0 4.8e+01 6.7e+03 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 592 1.0 2.3646e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 592 1.0 1.1952e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 2.0600e-07 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 295 1.0 4.9069e+00 1.5 0.00e+00 0.0 3.6e+03 7.9e+05 1.0e+00 14 0 4 53 0 14 0 4 53 0 0 MatGetOrdering 1 1.0 9.3876e-05 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatIncreaseOvrlp 1 1.0 1.9540e-03 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 295 1.0 2.3006e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 588 1.0 1.3316e-01 1.6 4.07e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 11516 KSPSetUp 296 1.0 1.1351e-03 9.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 295 1.0 1.1427e+01 1.9 1.23e+10 2.4 8.6e+04 2.7e+04 1.1e+04 39 72 90 43 47 39 72 90 43 47 2456 PCSetUp 590 1.0 8.6008e+00 2.0 5.19e+09 3.9 3.6e+03 7.9e+05 7.0e+00 21 24 4 53 0 21 24 4 53 0 1082 PCSetUpOnBlocks 295 1.0 3.7052e+00 3.7 5.19e+09 3.9 0.0e+00 0.0e+00 0.0e+00 7 24 0 0 0 7 24 0 0 0 2512 PCApply 3721 1.0 7.1734e+00 1.5 9.53e+09 3.7 4.5e+04 2.7e+04 1.0e+00 25 44 47 22 0 25 44 47 22 0 2416 PCApplyOnBlocks 3721 1.0 4.1748e+00 2.9 9.53e+09 3.7 0.0e+00 0.0e+00 0.0e+00 8 44 0 0 0 8 44 0 0 0 4152 ------------------------------------------------------------------------------------------------------------------------ From knepley at gmail.com Sun Jul 16 12:35:07 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 16 Jul 2023 13:35:07 -0400 Subject: [petsc-users] Bug in log_view printing In-Reply-To: <53873868-8b32-e5b2-c826-102ab448174e@gmail.com> References: <53873868-8b32-e5b2-c826-102ab448174e@gmail.com> Message-ID: On Sun, Jul 16, 2023 at 11:45?AM Jared Crean wrote: > Hello, > > I stumbled upon a small bug in the printing of the data when running > Petsc with -log_view. The entire table is attached, but here is an > example row: > Yes, we prescribe the exact number of spaces for each output, instead of allowing it to grow, so if the number exceeds the size, we lose our space. We wanted to keep the line length fixed. We do not recommend this form for processing. We usually use either the XML output for flame graphs since it does nesting, or the CSV output for processing with Pandas. Thanks, Matt > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) > Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSided 7 1.0 1.3282e-0271.6 0.00e+00 0.0 6.0e+01 4.0e+00 > 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > Notice how the Time Max and Time Ratio values are missing a space > separator (the other rows have only two digits for the exponents, I > think there should be a space after the e-02 and before the 7). > > Configuration details: > > * Petsc 3.17 > > * Arm64 architecture (and AWS c7g.xlarge instance, which use the > new Graviton3 processor) > > * ubuntu 22.04 > > * MPICH MPI (version 4.0, I think, whatever apt-get install mpich > gives) > > * clang-15 compiler > > > Jared Crean > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrean01 at gmail.com Sun Jul 16 12:38:43 2023 From: jcrean01 at gmail.com (Jared Crean) Date: Sun, 16 Jul 2023 11:38:43 -0600 Subject: [petsc-users] Bug in log_view printing In-Reply-To: References: <53873868-8b32-e5b2-c826-102ab448174e@gmail.com> Message-ID: <249b2830-ad27-2abc-0bde-d533a6c905b1@gmail.com> Neat, I didn't know about the XML output. ? Jared Crean On 7/16/2023 11:35 AM, Matthew Knepley wrote: > On Sun, Jul 16, 2023 at 11:45?AM Jared Crean wrote: > > Hello, > > ?? I stumbled upon a small bug in the printing of the data when > running > Petsc with -log_view.? The entire table is attached, but here is an > example row: > > > Yes, we prescribe the exact number of spaces for each output, instead > of allowing it to grow, > so if the number?exceeds the size, we lose our space. We wanted to > keep the line length fixed. > We do not recommend this form for processing. We usually use either > the XML output for flame > graphs since it does nesting, or the CSV output for processing with > Pandas. > > ? Thanks, > > ? ? ?Matt > > ------------------------------------------------------------------------------------------------------------------------ > Event??????????????? Count????? Time (sec) > Flop????????????????????????????? --- Global ---? --- Stage ---- Total > ??????????????????? Max Ratio? Max???? Ratio?? Max? Ratio Mess AvgLen > Reduct? %T %F %M %L %R? %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSided????????? 7 1.0 1.3282e-0271.6 0.00e+00 0.0 6.0e+01 > 4.0e+00 > 5.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0???? 0 > > > ?? Notice how the Time Max and Time Ratio values are missing a space > separator (the other rows have only two digits for the exponents, I > think there should be a space after the e-02 and before the 7). > > ?? Configuration details: > > ???? * Petsc 3.17 > > ???? * Arm64 architecture (and AWS c7g.xlarge instance, which use the > new Graviton3 processor) > > ???? * ubuntu 22.04 > > ???? * MPICH MPI (version 4.0, I think, whatever apt-get install > mpich > gives) > > ???? * clang-15 compiler > > > ?? Jared Crean > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cho at slac.stanford.edu Mon Jul 17 00:45:45 2023 From: cho at slac.stanford.edu (Ng, Cho-Kuen) Date: Mon, 17 Jul 2023 05:45:45 +0000 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: <10FFD366-3B5A-4B3D-A5AF-8BA0C093C882@petsc.dev> References: <10FFD366-3B5A-4B3D-A5AF-8BA0C093C882@petsc.dev> Message-ID: Barry, Thank you so much for the clarification. I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are there other tutorials available? Cho ________________________________ From: Barry Smith Sent: Saturday, July 15, 2023 8:36 AM To: Ng, Cho-Kuen Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Using PETSc GPU backend Cho, We currently have a crappy API for turning on GPU support, and our documentation is misleading in places. People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect. This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda If you use DM to create the matrices and vectors then you can use -dm_mat_type aijcusparse -dm_vec_type cuda Sorry for the confusion. Barry On Jul 15, 2023, at 8:03 AM, Matthew Knepley wrote: On Sat, Jul 15, 2023 at 1:44?AM Ng, Cho-Kuen > wrote: Matt, After inserting 2 lines in the code: ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); "There are no unused options." However, there is no improvement on the GPU performance. 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual. 2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions(). THanks, Matt Thanks, Cho ________________________________ From: Matthew Knepley > Sent: Friday, July 14, 2023 5:57 PM To: Ng, Cho-Kuen > Cc: Barry Smith >; Mark Adams >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen > wrote: I managed to pass the following options to PETSc using a GPU node on Perlmutter. -mat_type aijcusparse -vec_type cuda -log_view -options_left Below is a summary of the test using 4 MPI tasks and 1 GPU per task. o #PETSc Option Table entries: ???-log_view ???-mat_type aijcusparse -options_left -vec_type cuda #End of PETSc Option Table entries WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! There is one unused database option. It is: Option left: name:-mat_type value: aijcusparse The -mat_type option has not been used. In the application code, we use ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line. Thanks, Matt o The percent flops on the GPU for KSPSolve is 17%. In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance? Thanks, Cho ________________________________ From: Ng, Cho-Kuen > Sent: Friday, June 30, 2023 7:57 AM To: Barry Smith >; Mark Adams > Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Barry, Mark and Matt, Thank you all for the suggestions. I will modify the code so we can pass runtime options. Cho ________________________________ From: Barry Smith > Sent: Friday, June 30, 2023 7:01 AM To: Mark Adams > Cc: Matthew Knepley >; Ng, Cho-Kuen >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know. On Jun 30, 2023, at 9:30 AM, Mark Adams > wrote: PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run: a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left Mark On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley > wrote: On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users > wrote: Mark, The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right? PETSc will read command line argument automatically in PetscInitialize() unless you shut it off. Thanks, Matt Cho ________________________________ From: Ng, Cho-Kuen > Sent: Thursday, June 29, 2023 8:32 PM To: Mark Adams > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Mark, Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options? Cho ________________________________ From: Mark Adams > Sent: Thursday, June 29, 2023 5:55 PM To: Ng, Cho-Kuen > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Using PETSc GPU backend Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0. The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left. Mark On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users > wrote: I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? Thanks, Cho -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From quentin.chevalier at polytechnique.edu Mon Jul 17 04:48:55 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Mon, 17 Jul 2023 11:48:55 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: Message-ID: Hello Jose, I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. Thank you for your time, Quentin On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > > By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > > An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > > Jose > > > > El 12 jul 2023, a las 19:04, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > > > > Hello PETSc Users, > > > > I have a generalised eigenvalue problem : Ax= lambda Bx > > I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > > > Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > > > I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > > > A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > > > I use PETSc in complex mode through the petsc4py bridge. > > > > Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > > > Thank you for your time, > > > > Quentin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Jul 17 04:56:54 2023 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Jul 2023 11:56:54 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: Message-ID: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. Jose > El 17 jul 2023, a las 11:48, Quentin Chevalier escribi?: > > Hello Jose, > > I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. > > I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. > > I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. > > Thank you for your time, > > Quentin > > > On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > > > > By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > > > > An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > > > > Jose > > > > > > > El 12 jul 2023, a las 19:04, Quentin Chevalier escribi?: > > > > > > Hello PETSc Users, > > > > > > I have a generalised eigenvalue problem : Ax= lambda Bx > > > I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > > > > > Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > > > > > I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > > > > > A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > > > > > I use PETSc in complex mode through the petsc4py bridge. > > > > > > Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > > > > > Thank you for your time, > > > > > > Quentin > > From itugrik76 at mail.ru Mon Jul 17 05:07:33 2023 From: itugrik76 at mail.ru (=?UTF-8?B?0JrQvtC90YHRgtCw0L3RgtC40L0=?=) Date: Mon, 17 Jul 2023 13:07:33 +0300 Subject: [petsc-users] =?utf-8?q?windows_build?= In-Reply-To: <494bbb14-4371-7e64-c911-76786f8b5f8f@mcs.anl.gov> References: <1689105321.125562771@f340.i.mail.ru> <494bbb14-4371-7e64-c911-76786f8b5f8f@mcs.anl.gov> Message-ID: <1689588453.388454834@f307.i.mail.ru> Hello. I have such problem, while I'm making petsc $ make C:/cygwin64/home/itugr/asd/petsc//lib/petsc/conf/petscvariables:140: *** Too many open files. ?Stop. ?Are there any advices how to fix it? ? ? -- ?????????? ? ? >???????, 11 ???? 2023, 23:09 +03:00 ?? Satish Balay : >? >On Tue, 11 Jul 2023, ?????????? via petsc-users wrote: >? >> >> Hello, I'm trying to build petsc on windows. And when I make it I have such problem >from the screenshot - it appears you are not using cygwin terminal(bash-shell) - as per instructions > >https://petsc.org/release/install/windows/ > >What compilers are you attempting to use with PETSc? Can you retry your build - as per the above instructions? > >If you still encounter issues - send us the corresponding configure.log. > >Also - best if you can copy/paste text from terminal - instead of screenshots > >Note: if you can use WSL2(linux) on your windows machine - that might be an easier install > >Satish ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From badi.hamid at gmail.com Mon Jul 17 05:12:11 2023 From: badi.hamid at gmail.com (hamid badi) Date: Mon, 17 Jul 2023 12:12:11 +0200 Subject: [petsc-users] windows build In-Reply-To: <1689588453.388454834@f307.i.mail.ru> References: <1689105321.125562771@f340.i.mail.ru> <494bbb14-4371-7e64-c911-76786f8b5f8f@mcs.anl.gov> <1689588453.388454834@f307.i.mail.ru> Message-ID: For me, I got no problem when compiling Petsc with Mumps under mingw/clang or gcc, everything works fine. May be i could write a tutorial when I'll have time, now i'm on holidays. Le lun. 17 juil. 2023 ? 12:08, ?????????? via petsc-users < petsc-users at mcs.anl.gov> a ?crit : > Hello. I have such problem, while I'm making petsc > $ make > C:/cygwin64/home/itugr/asd/petsc//lib/petsc/conf/petscvariables:140: *** > Too many open files. Stop. > Are there any advices how to fix it? > > > -- > ?????????? > > > > ???????, 11 ???? 2023, 23:09 +03:00 ?? Satish Balay : > > On Tue, 11 Jul 2023, ?????????? via petsc-users wrote: > > > > > Hello, I'm trying to build petsc on windows. And when I make it I have > such problem > > from the screenshot - it appears you are not using cygwin > terminal(bash-shell) - as per instructions > > https://petsc.org/release/install/windows/ > > What compilers are you attempting to use with PETSc? Can you retry your > build - as per the above instructions? > > If you still encounter issues - send us the corresponding configure.log. > > Also - best if you can copy/paste text from terminal - instead of > screenshots > > Note: if you can use WSL2(linux) on your windows machine - that might be > an easier install > > Satish > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 05:42:44 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 06:42:44 -0400 Subject: [petsc-users] windows build In-Reply-To: <1689588453.388454834@f307.i.mail.ru> References: <1689105321.125562771@f340.i.mail.ru> <494bbb14-4371-7e64-c911-76786f8b5f8f@mcs.anl.gov> <1689588453.388454834@f307.i.mail.ru> Message-ID: On Mon, Jul 17, 2023 at 6:08?AM ?????????? via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello. I have such problem, while I'm making petsc > $ make > C:/cygwin64/home/itugr/asd/petsc//lib/petsc/conf/petscvariables:140: *** > Too many open files. Stop. > Are there any advices how to fix it? > This is an OS problem, and since this is Windows, I would try rebooting. THanks, Matt > -- > ?????????? > > > > ???????, 11 ???? 2023, 23:09 +03:00 ?? Satish Balay : > > On Tue, 11 Jul 2023, ?????????? via petsc-users wrote: > > > > > Hello, I'm trying to build petsc on windows. And when I make it I have > such problem > > from the screenshot - it appears you are not using cygwin > terminal(bash-shell) - as per instructions > > https://petsc.org/release/install/windows/ > > What compilers are you attempting to use with PETSc? Can you retry your > build - as per the above instructions? > > If you still encounter issues - send us the corresponding configure.log. > > Also - best if you can copy/paste text from terminal - instead of > screenshots > > Note: if you can use WSL2(linux) on your windows machine - that might be > an easier install > > Satish > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From quentin.chevalier at polytechnique.edu Mon Jul 17 08:50:54 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Mon, 17 Jul 2023 15:50:54 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> References: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> Message-ID: Thank you for this suggestion, I tried to implement that but it's proven pretty hard to implement MATOP_GET_DIAGONAL without completely tanking performance. After all, B is a shell matrix for a reason : it looks like M+R^H P M P R with R itself a shell matrix. Allow me to point out that I have no shift. My eigenvalue problem is purely about the largest ones out there. Section 8.2 and 3.4.3 led me to think that there was a way to avoid computing (or writing a shell matrix about it) B^-1... But you seem to stress that there's no way around it. Quentin On Mon, 17 Jul 2023 at 11:56, Jose E. Roman wrote: > > The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. > > If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. > > Jose > > > > El 17 jul 2023, a las 11:48, Quentin Chevalier escribi?: > > > > Hello Jose, > > > > I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. > > > > I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. > > > > I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. > > > > Thank you for your time, > > > > Quentin > > > > > > On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > > > > > > By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > > > > > > An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > > > > > > Jose > > > > > > > > > > El 12 jul 2023, a las 19:04, Quentin Chevalier escribi?: > > > > > > > > Hello PETSc Users, > > > > > > > > I have a generalised eigenvalue problem : Ax= lambda Bx > > > > I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > > > > > > > Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > > > > > > > I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > > > > > > > A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > > > > > > > I use PETSc in complex mode through the petsc4py bridge. > > > > > > > > Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > > > > > > > Thank you for your time, > > > > > > > > Quentin > > > > From bsmith at petsc.dev Mon Jul 17 08:58:15 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 17 Jul 2023 09:58:15 -0400 Subject: [petsc-users] Using PETSc GPU backend In-Reply-To: References: <10FFD366-3B5A-4B3D-A5AF-8BA0C093C882@petsc.dev> Message-ID: The examples that use DM, in particular DMDA all trivially support using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda > On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen wrote: > > Barry, > > Thank you so much for the clarification. > > I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are there other tutorials available? > > Cho > From: Barry Smith > > Sent: Saturday, July 15, 2023 8:36 AM > To: Ng, Cho-Kuen > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Using PETSc GPU backend > > > > Cho, > > We currently have a crappy API for turning on GPU support, and our documentation is misleading in places. > > People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect. > > This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda > > If you use DM to create the matrices and vectors then you can use -dm_mat_type aijcusparse -dm_vec_type cuda > > Sorry for the confusion. > > Barry > > > > >> On Jul 15, 2023, at 8:03 AM, Matthew Knepley > wrote: >> >> On Sat, Jul 15, 2023 at 1:44?AM Ng, Cho-Kuen > wrote: >> Matt, >> >> After inserting 2 lines in the code: >> >> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> ierr = MatSetFromOptions(A);CHKERRQ(ierr); >> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >> "There are no unused options." However, there is no improvement on the GPU performance. >> >> 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual. >> >> 2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions(). >> >> THanks, >> >> Matt >> >> Thanks, >> Cho >> >> From: Matthew Knepley > >> Sent: Friday, July 14, 2023 5:57 PM >> To: Ng, Cho-Kuen > >> Cc: Barry Smith >; Mark Adams >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> On Fri, Jul 14, 2023 at 7:57?PM Ng, Cho-Kuen > wrote: >> I managed to pass the following options to PETSc using a GPU node on Perlmutter. >> >> -mat_type aijcusparse -vec_type cuda -log_view -options_left >> >> Below is a summary of the test using 4 MPI tasks and 1 GPU per task. >> >> o #PETSc Option Table entries: >> ???-log_view >> ???-mat_type aijcusparse >> -options_left >> -vec_type cuda >> #End of PETSc Option Table entries >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> There is one unused database option. It is: >> Option left: name:-mat_type value: aijcusparse >> >> The -mat_type option has not been used. In the application code, we use >> >> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n, >> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr); >> >> >> If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line. >> >> Thanks, >> >> Matt >> >> o The percent flops on the GPU for KSPSolve is 17%. >> >> In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance? >> >> Thanks, >> Cho >> From: Ng, Cho-Kuen > >> Sent: Friday, June 30, 2023 7:57 AM >> To: Barry Smith >; Mark Adams > >> Cc: Matthew Knepley >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> Barry, Mark and Matt, >> >> Thank you all for the suggestions. I will modify the code so we can pass runtime options. >> >> Cho >> From: Barry Smith > >> Sent: Friday, June 30, 2023 7:01 AM >> To: Mark Adams > >> Cc: Matthew Knepley >; Ng, Cho-Kuen >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Using PETSc GPU backend >> >> >> Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know. >> >>> On Jun 30, 2023, at 9:30 AM, Mark Adams > wrote: >>> >>> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run: >>> >>> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left >>> >>> Mark >>> >>> On Fri, Jun 30, 2023 at 6:16?AM Matthew Knepley > wrote: >>> On Fri, Jun 30, 2023 at 1:13?AM Ng, Cho-Kuen via petsc-users > wrote: >>> Mark, >>> >>> The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right? >>> >>> PETSc will read command line argument automatically in PetscInitialize() unless you shut it off. >>> >>> Thanks, >>> >>> Matt >>> >>> Cho >>> From: Ng, Cho-Kuen > >>> Sent: Thursday, June 29, 2023 8:32 PM >>> To: Mark Adams > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Mark, >>> >>> Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options? >>> >>> Cho >>> From: Mark Adams > >>> Sent: Thursday, June 29, 2023 5:55 PM >>> To: Ng, Cho-Kuen > >>> Cc: petsc-users at mcs.anl.gov > >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left >>> >>> The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0. >>> >>> The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left. >>> >>> Mark >>> >>> On Thu, Jun 29, 2023 at 7:50?PM Ng, Cho-Kuen via petsc-users > wrote: >>> I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? >>> >>> Thanks, >>> Cho >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From philliprusso at proton.me Sat Jul 15 23:26:00 2023 From: philliprusso at proton.me (philliprusso) Date: Sun, 16 Jul 2023 04:26:00 +0000 Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? Message-ID: I cloned petsc source for use with dolfinx project. So after .configure mak sudo make install I found there was some type of difficulty with the destination directory so I copied the files manually into usr/includes of Ubuntu 22.04 jammy. So some petsc header files are now found for compiling cpp dolfinx source code but petscconf.h still not found by dolfinxs source tree of header files. Anyone know how to remedy this so dolfinx cpps can find petsscconf.h that g++ is claiming as missing? Thank you! Sent with [Proton Mail](https://proton.me/) secure email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 09:58:24 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 10:58:24 -0400 Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? In-Reply-To: References: Message-ID: On Mon, Jul 17, 2023 at 10:55?AM philliprusso via petsc-users < petsc-users at mcs.anl.gov> wrote: > I cloned petsc source for use with dolfinx project. So after .configure > mak sudo make install I found there was some type of difficulty with the > destination directory so I copied the files manually into usr/includes of > Ubuntu 22.04 jammy. So some petsc header files are now found for compiling > cpp dolfinx source code but petscconf.h still not found by dolfinxs source > tree of header files. Anyone know how to remedy this so dolfinx cpps can > find petsscconf.h that g++ is claiming as missing? Thank you! > 1. I think the safest thing is to fix the install 2. petscconf.h is in $PETSC_ARCH/include, not the top-level include, since it is build-specific Thanks, Matt > Sent with Proton Mail secure email. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jul 17 10:08:29 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 17 Jul 2023 10:08:29 -0500 (CDT) Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? In-Reply-To: References: Message-ID: We do not recommend installing PETSc in /usr/include [aka --prefix=/usr]. Now you have some petsc includes here that will potentially conflict with any other install of PETSc you might attempt. You might have to manually check - and delete PETSc installed files from here. For most uses - one can install PETSc in $HOME If you need a common install - suggest --prefix=/usr/local/petsc-version Satish On Sun, 16 Jul 2023, philliprusso via petsc-users wrote: > I cloned petsc source for use with dolfinx project. So after .configure mak sudo make install I found there was some type of difficulty with the destination directory so I copied the files manually into usr/includes of Ubuntu 22.04 jammy. So some petsc header files are now found for compiling cpp dolfinx source code but petscconf.h still not found by dolfinxs source tree of header files. Anyone know how to remedy this so dolfinx cpps can find petsscconf.h that g++ is claiming as missing? Thank you! > > Sent with [Proton Mail](https://proton.me/) secure email. From jroman at dsic.upv.es Mon Jul 17 10:29:24 2023 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Jul 2023 17:29:24 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> Message-ID: It is possible to pass a different matrix to build the preconditioner. That is, the shell matrix for B (EPSSetOperators) and an explicit matrix (that approximates B) for the preconditioner. For instance, you can try passing M for building the preconditioner. Since M is an explicit matrix, you can try the default preconditioner (block Jacobi with ILU as local solver) or even a full LU decomposition. The effectiveness of the preconditioner will depend on how the update M+R^H P M P R moves the eigenvalues around. You can do this with STSetSplitPreconditioner() or STSetPreconditionerMat(). In your case any of them will do. Jose > El 17 jul 2023, a las 15:50, Quentin Chevalier escribi?: > > Thank you for this suggestion, I tried to implement that but it's > proven pretty hard to implement MATOP_GET_DIAGONAL without completely > tanking performance. After all, B is a shell matrix for a reason : it > looks like M+R^H P M P R with R itself a shell matrix. > > Allow me to point out that I have no shift. My eigenvalue problem is > purely about the largest ones out there. Section 8.2 and 3.4.3 led me > to think that there was a way to avoid computing (or writing a shell > matrix about it) B^-1... But you seem to stress that there's no way > around it. > > Quentin > > > > On Mon, 17 Jul 2023 at 11:56, Jose E. Roman wrote: >> >> The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. >> >> If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. >> >> Jose >> >> >>> El 17 jul 2023, a las 11:48, Quentin Chevalier escribi?: >>> >>> Hello Jose, >>> >>> I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. >>> >>> I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. >>> >>> I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. >>> >>> Thank you for your time, >>> >>> Quentin >>> >>> >>> On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: >>>> >>>> By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). >>>> >>>> An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. >>>> >>>> Jose >>>> >>>> >>>>> El 12 jul 2023, a las 19:04, Quentin Chevalier escribi?: >>>>> >>>>> Hello PETSc Users, >>>>> >>>>> I have a generalised eigenvalue problem : Ax= lambda Bx >>>>> I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. >>>>> >>>>> Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. >>>>> >>>>> I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. >>>>> >>>>> A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. >>>>> >>>>> I use PETSc in complex mode through the petsc4py bridge. >>>>> >>>>> Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. >>>>> >>>>> Thank you for your time, >>>>> >>>>> Quentin >>>> >> From miguel.salazar at corintis.com Mon Jul 17 11:42:10 2023 From: miguel.salazar at corintis.com (Miguel Angel Salazar de Troya) Date: Mon, 17 Jul 2023 18:42:10 +0200 Subject: [petsc-users] Structured (DMDA) vs Unstructured (DMPlex) meshes Message-ID: Hello, I am trying to understand if I should make the effort to make my code use structured meshes instead of unstructured ones. My domain is cartesian so that is the first check for structured meshes. However, the problem size I am looking at is ~20 million degrees of freedom. My understanding is that for this problem size, most of the time is spent on the solver. In this case, do structured meshes still have an advantage? Can they run Krylov methods faster than when using structured meshes? What about other solvers and preconditioners? Thanks, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 17 11:48:36 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 17 Jul 2023 12:48:36 -0400 Subject: [petsc-users] Structured (DMDA) vs Unstructured (DMPlex) meshes In-Reply-To: References: Message-ID: <88F24FD0-4650-4B61-8FB4-C9B1EDB3FF9A@petsc.dev> The largest potential advantage of DMDA is likely the possibility of easily using geometric multigrid if it is appropriate for the problem (or subproblem of the problem) you are solving. The second advantage is, this depends on your PDE and discretization, the simplicity of your code, and what part of PETSc it depends on, if it doesn't depend on DMPLEX (and doesn't need to) then that is a good thing. Disadvantage of DMDA is lack of flexibility in discretization, handling of non-standard boundary conditions, geometry. > On Jul 17, 2023, at 12:42 PM, Miguel Angel Salazar de Troya wrote: > > Hello, > > I am trying to understand if I should make the effort to make my code use structured meshes instead of unstructured ones. My domain is cartesian so that is the first check for structured meshes. However, the problem size I am looking at is ~20 million degrees of freedom. My understanding is that for this problem size, most of the time is spent on the solver. In this case, do structured meshes still have an advantage? Can they run Krylov methods faster than when using structured meshes? What about other solvers and preconditioners? > > Thanks, > Miguel From karthikeyan.chockalingam at stfc.ac.uk Mon Jul 17 12:13:59 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 17 Jul 2023 17:13:59 +0000 Subject: [petsc-users] Parallel matrix multiplication Message-ID: Hello, I would like to perform the following operation [P^T A P] x? = P^T f Where P is a rectangular matrix and A is a square matrix. All the matrixes are constructed using MPIAIJ. Should I be concerned about the parallel partitioning of the matrix P and A? Or can I just go ahead and use MatMatMult to compute [P^T A P]? Is there any in-built functionality to perform this operation? Kind regards, Karthik. -- Dr. Karthik Chockalingam High-Performance Software Engineer Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From FERRANJ2 at my.erau.edu Mon Jul 17 12:51:24 2023 From: FERRANJ2 at my.erau.edu (Ferrand, Jesus A.) Date: Mon, 17 Jul 2023 17:51:24 +0000 Subject: [petsc-users] PETSc Installation Assistance Message-ID: Greetings. I recently changed operating systems (Ubuntu 20.04 -> Debian 12 "Bookworm") and tried to reinstall PETSc. I tried doing the usual as described in (https://petsc.org/release/install/download/#recommended-obtain-release-version-with-git): * git clone/pull * ./configure -- ... -- * make install * make check Everything proceeds smoothly until the "make check" step, where I get the following error: ============================================================================== Running check examples to verify correct installation Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and PETSC_ARCH=arch-linux-c-debug Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 0a1,3 > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/tests Possible problem with ex47 running with hdf5, diffs above ========================================= Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). Number of SNES iterations = 3 Completed test examples Error while running make check gmake[1]: *** [makefile:123: check] Error 1 make: *** [GNUmakefile:17: check] Error 2 ============================================================================== I tried reinstalling the same version I was able to use prior to changing OS's (PETSc 3.18.3, via tarball) and get a similar error. ============================================================================== make PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_P arty/newPETSC/petsc-3.18.3 PETSC_ARCH=arch-linux-c-debug check Running check examples to verify correct installation Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3 and PETSC_ARCH=arch-linux-c-debug Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 0a1,3 > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). /home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3/src/vec/vec/tests Possible problem with ex47 running with hdf5, diffs above ========================================= Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). Number of SNES iterations = 3 Completed test examples Error while running make check gmake[1]: *** [makefile:149: check] Error 1 make: *** [GNUmakefile:17: check] Error 2 ============================================================================== I checked the FAQs (https://petsc.org/release/faq/#table-of-contents) and tried the suggestions there (specifically the points under https://petsc.org/release/faq/#what-does-it-mean-when-make-check-hangs-or-errors-on-petscoptionsinsertfile) to no avail. * I don't use a VPN. * I tried running make check with Wifi disabled. * Ran ./configure with --download-mpich-device=ch3:nemesis --download-mpich Surprisingly, I can compile my PETSc programs using a makefile, and run them. However, the programs are now broken (I get SIGSEGV originating from within DM/DMPlex APIs). Sincerely: J.A. Ferrand Embry-Riddle Aeronautical University - Daytona Beach - FL Ph.D. Candidate, Aerospace Engineering M.Sc. Aerospace Engineering B.Sc. Aerospace Engineering B.Sc. Computational Mathematics Phone: (386)-843-1829 Email(s): ferranj2 at my.erau.edu jesus.ferrand at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.jolivet at lip6.fr Mon Jul 17 12:58:13 2023 From: pierre.jolivet at lip6.fr (Pierre Jolivet) Date: Mon, 17 Jul 2023 19:58:13 +0200 Subject: [petsc-users] PETSc Installation Assistance In-Reply-To: References: Message-ID: <28151F30-4A69-4E37-9699-794F1ECB8F40@lip6.fr> https://petsc.org/release/faq/#what-does-the-message-hwloc-linux-ignoring-pci-device-with-non-16bit-domain-mean Thanks, Pierre > On 17 Jul 2023, at 7:51 PM, Ferrand, Jesus A. wrote: > > Greetings. > > I recently changed operating systems (Ubuntu 20.04 -> Debian 12 "Bookworm") and tried to reinstall PETSc. > I tried doing the usual as described in (https://petsc.org/release/install/download/#recommended-obtain-release-version-with-git): > git clone/pull > ./configure -- ... -- > make install > make check > Everything proceeds smoothly until the "make check" step, where I get the following error: > ============================================================================== > Running check examples to verify correct installation > Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and PETSC_ARCH=arch-linux-c-debug > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > 0a1,3 > > hwloc/linux: Ignoring PCI device with non-16bit domain. > > Pass --enable-32bits-pci-domain to configure to support such devices > > (warning: it would break the library ABI, don't enable unless really needed). > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/tests > Possible problem with ex47 running with hdf5, diffs above > ========================================= > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > Number of SNES iterations = 3 > Completed test examples > Error while running make check > gmake[1]: *** [makefile:123: check] Error 1 > make: *** [GNUmakefile:17: check] Error 2 > ============================================================================== > > > I tried reinstalling the same version I was able to use prior to changing OS's (PETSc 3.18.3, via tarball) and get a similar error. > ============================================================================== > make PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_P > arty/newPETSC/petsc-3.18.3 PETSC_ARCH=arch-linux-c-debug check > Running check examples to verify correct installation > Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3 and PETSC_ARCH=arch-linux-c-debug > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > 0a1,3 > > hwloc/linux: Ignoring PCI device with non-16bit domain. > > Pass --enable-32bits-pci-domain to configure to support such devices > > (warning: it would break the library ABI, don't enable unless really needed). > /home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3/src/vec/vec/tests > Possible problem with ex47 running with hdf5, diffs above > ========================================= > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really needed). > Number of SNES iterations = 3 > Completed test examples > Error while running make check > gmake[1]: *** [makefile:149: check] Error 1 > make: *** [GNUmakefile:17: check] Error 2 > ============================================================================== > > > I checked the FAQs (https://petsc.org/release/faq/#table-of-contents) and tried the suggestions there (specifically the points underhttps://petsc.org/release/faq/#what-does-it-mean-when-make-check-hangs-or-errors-on-petscoptionsinsertfile) to no avail. > I don't use a VPN. > I tried running make check with Wifi disabled. > Ran ./configure with --download-mpich-device=ch3:nemesis --download-mpich > > Surprisingly, I can compile my PETSc programs using a makefile, and run them. > However, the programs are now broken (I get SIGSEGV originating from within DM/DMPlex APIs). > > > Sincerely: > > J.A. Ferrand > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > M.Sc. Aerospace Engineering > B.Sc. Aerospace Engineering > B.Sc. Computational Mathematics > > Phone: (386)-843-1829 > Email(s): ferranj2 at my.erau.edu > jesus.ferrand at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From philliprusso at proton.me Mon Jul 17 13:06:39 2023 From: philliprusso at proton.me (philliprusso) Date: Mon, 17 Jul 2023 18:06:39 +0000 Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? In-Reply-To: References: Message-ID: So I have this to go by: export PETSC_DIR=/absolute/path/to/petsc export PETSC_ARCH=linux-gnu-c-debug PETSC_DIR Is that the installation/destination path or the path where the source code is? I feel confused.... So there is a destination folder that dolfinx project wnats for petsc. I have an idea it might want /usr/local/include not sure of everything quite yet... Sent with Proton Mail secure email. ------- Original Message ------- On Monday, July 17th, 2023 at 8:08 AM, Satish Balay wrote: > We do not recommend installing PETSc in /usr/include [aka --prefix=/usr]. Now you have some petsc includes here that will potentially conflict with any other install of PETSc you might attempt. > > You might have to manually check - and delete PETSc installed files from here. > > For most uses - one can install PETSc in $HOME > > If you need a common install - suggest --prefix=/usr/local/petsc-version > > Satish > > On Sun, 16 Jul 2023, philliprusso via petsc-users wrote: > > > I cloned petsc source for use with dolfinx project. So after .configure mak sudo make install I found there was some type of difficulty with the destination directory so I copied the files manually into usr/includes of Ubuntu 22.04 jammy. So some petsc header files are now found for compiling cpp dolfinx source code but petscconf.h still not found by dolfinxs source tree of header files. Anyone know how to remedy this so dolfinx cpps can find petsscconf.h that g++ is claiming as missing? Thank you! > > > > Sent with Proton Mail secure email. From bsmith at petsc.dev Mon Jul 17 13:21:45 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 17 Jul 2023 14:21:45 -0400 Subject: [petsc-users] Parallel matrix multiplication In-Reply-To: References: Message-ID: https://petsc.org/release/manualpages/Mat/MatPtAP/ also note that PETSc has a large infrastructure for efficient ways to compute various matrix-matrix operations with a variety of algorithms that can be all accessed by starting with https://petsc.org/release/manualpages/Mat/MatProductCreate/ > On Jul 17, 2023, at 1:13 PM, Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > > Hello, > > I would like to perform the following operation > > [P^T A P] x? = P^T f > > Where P is a rectangular matrix and A is a square matrix. All the matrixes are constructed using MPIAIJ. > > Should I be concerned about the parallel partitioning of the matrix P and A? > > Or can I just go ahead and use MatMatMult to compute [P^T A P]? > > Is there any in-built functionality to perform this operation? > > Kind regards, > Karthik. > > > > > > > > -- > Dr. Karthik Chockalingam > High-Performance Software Engineer > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 17 13:26:10 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 17 Jul 2023 14:26:10 -0400 Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? In-Reply-To: References: Message-ID: When configuring and making PETSc, PETSC_DIR can be empty or point to the directory with the PETSc source If you used --prefix to configure and install PETSc then When using PETSc to compile other source code and using a makefile that utilizes PETSC_DIR, then PETSC_DIR needs to point to the --prefix location PETSC_ARCH should be empty since it is not used for --prefix installs > On Jul 17, 2023, at 2:06 PM, philliprusso via petsc-users wrote: > > So I have this to go by: > export PETSC_DIR=/absolute/path/to/petsc > > export PETSC_ARCH=linux-gnu-c-debug > > PETSC_DIR Is that the installation/destination path or the path where the source code is? I feel confused.... > > So there is a destination folder that dolfinx project wnats for petsc. I have an idea it might want /usr/local/include not sure of everything quite yet... > > > > > Sent with Proton Mail secure email. > > ------- Original Message ------- > On Monday, July 17th, 2023 at 8:08 AM, Satish Balay wrote: > > >> We do not recommend installing PETSc in /usr/include [aka --prefix=/usr]. Now you have some petsc includes here that will potentially conflict with any other install of PETSc you might attempt. >> >> You might have to manually check - and delete PETSc installed files from here. >> >> For most uses - one can install PETSc in $HOME >> >> If you need a common install - suggest --prefix=/usr/local/petsc-version >> >> Satish >> >> On Sun, 16 Jul 2023, philliprusso via petsc-users wrote: >> >>> I cloned petsc source for use with dolfinx project. So after .configure mak sudo make install I found there was some type of difficulty with the destination directory so I copied the files manually into usr/includes of Ubuntu 22.04 jammy. So some petsc header files are now found for compiling cpp dolfinx source code but petscconf.h still not found by dolfinxs source tree of header files. Anyone know how to remedy this so dolfinx cpps can find petsscconf.h that g++ is claiming as missing? Thank you! >>> >>> Sent with Proton Mail secure email. From balay at mcs.anl.gov Mon Jul 17 13:39:09 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 17 Jul 2023 13:39:09 -0500 (CDT) Subject: [petsc-users] petscconf.h missing building cpp file with dolfinx? In-Reply-To: References: Message-ID: <77aa1f72-ccc9-abff-ae4f-7bc7593ad123@mcs.anl.gov> PETSc supports both --prefix and inplace install. Suggest using inplace install - in your $HOME: For ex: >>> balay at p1 /home/balay $ tar -xzf petsc-3.19. petsc-3.19.2.tar.gz petsc-3.19.3.tar.gz balay at p1 /home/balay $ tar -xzf petsc-3.19.3.tar.gz balay at p1 /home/balay $ cd petsc-3.19.3 balay at p1 /home/balay/petsc-3.19.3 $ export PETSC_DIR=$PWD balay at p1 /home/balay/petsc-3.19.3 $ export PETSC_ARCH=arch-buildfordolfin balay at p1 /home/balay/petsc-3.19.3 $ ./configure && make <<< [with the required configure options to get a successful build of PETSc] Now - with PETSC_DIR and PETSC_ARCH (env variables) - set to the above values - try installing dolfin Satish On Mon, 17 Jul 2023, Barry Smith wrote: > > When configuring and making PETSc, PETSC_DIR can be empty or point to the directory with the PETSc source > > If you used --prefix to configure and install PETSc then > > When using PETSc to compile other source code and using a makefile that utilizes PETSC_DIR, then PETSC_DIR needs to point to the --prefix location > > PETSC_ARCH should be empty since it is not used for --prefix installs > > > > > On Jul 17, 2023, at 2:06 PM, philliprusso via petsc-users wrote: > > > > So I have this to go by: > > export PETSC_DIR=/absolute/path/to/petsc > > > > export PETSC_ARCH=linux-gnu-c-debug > > > > PETSC_DIR Is that the installation/destination path or the path where the source code is? I feel confused.... > > > > So there is a destination folder that dolfinx project wnats for petsc. I have an idea it might want /usr/local/include not sure of everything quite yet... > > > > > > > > > > Sent with Proton Mail secure email. > > > > ------- Original Message ------- > > On Monday, July 17th, 2023 at 8:08 AM, Satish Balay wrote: > > > > > >> We do not recommend installing PETSc in /usr/include [aka --prefix=/usr]. Now you have some petsc includes here that will potentially conflict with any other install of PETSc you might attempt. > >> > >> You might have to manually check - and delete PETSc installed files from here. > >> > >> For most uses - one can install PETSc in $HOME > >> > >> If you need a common install - suggest --prefix=/usr/local/petsc-version > >> > >> Satish > >> > >> On Sun, 16 Jul 2023, philliprusso via petsc-users wrote: > >> > >>> I cloned petsc source for use with dolfinx project. So after .configure mak sudo make install I found there was some type of difficulty with the destination directory so I copied the files manually into usr/includes of Ubuntu 22.04 jammy. So some petsc header files are now found for compiling cpp dolfinx source code but petscconf.h still not found by dolfinxs source tree of header files. Anyone know how to remedy this so dolfinx cpps can find petsscconf.h that g++ is claiming as missing? Thank you! > >>> > >>> Sent with Proton Mail secure email. > From balay at mcs.anl.gov Mon Jul 17 14:05:51 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 17 Jul 2023 14:05:51 -0500 (CDT) Subject: [petsc-users] PETSc Installation Assistance In-Reply-To: <28151F30-4A69-4E37-9699-794F1ECB8F40@lip6.fr> References: <28151F30-4A69-4E37-9699-794F1ECB8F40@lip6.fr> Message-ID: <0cb156cb-18d8-a9f0-2650-7473a6473cc3@mcs.anl.gov> On Mon, 17 Jul 2023, Pierre Jolivet wrote: > https://petsc.org/release/faq/#what-does-the-message-hwloc-linux-ignoring-pci-device-with-non-16bit-domain-mean > > On 17 Jul 2023, at 7:51 PM, Ferrand, Jesus A. wrote: > > hwloc/linux: Ignoring PCI device with non-16bit domain. > > Pass --enable-32bits-pci-domain to configure to support such devices > > (warning: it would break the library ABI, don't enable unless really needed). > > Ran ./configure with --download-mpich-device=ch3:nemesis --download-mpich > > > > Surprisingly, I can compile my PETSc programs using a makefile, and run them. > > However, the programs are now broken (I get SIGSEGV originating from within DM/DMPlex APIs). This must be a different issue - might need to run in debugger or valgrind to check this failure. Satish From pierre.jolivet at lip6.fr Mon Jul 17 14:11:21 2023 From: pierre.jolivet at lip6.fr (Pierre Jolivet) Date: Mon, 17 Jul 2023 21:11:21 +0200 Subject: [petsc-users] [EXTERNAL] PETSc Installation Assistance In-Reply-To: References: <28151F30-4A69-4E37-9699-794F1ECB8F40@lip6.fr> Message-ID: > On 17 Jul 2023, at 9:00 PM, Ferrand, Jesus A. wrote: > > Pierre: > Setting the environment variable allows make check to complete without errors. > It only seems to run three checks. > I recall that in the past, make check would run 100+ tests. You are probably thinking of make test, which is running thousands of tests indeed. > This is what I get from make check: > ========================================================================= > Running check examples to verify correct installation > Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and PETSC_ARCH=arch-linux-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/vec/vec/tests/ex47 run successfully with hdf5 > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > ========================================================================= > > Anyhow, some of the APIs that worked before now break. > For example "DMPlexCreateGmshFromFile()" breaks when I read simple test mesh. > I tried configuring and installing both the newest release of PETSc and that tarball of version 3.18.3. > Both seem to be breaking APIs that used to work in the older OS. There has been some internal changes to the behavior of DMPlex, they should be documented in the change log. But you are saying that with the same version, just by switching OS, you get a different behavior? Do as Satish suggested then, because it?s unlikely that the root cause is in the library itself. Thanks, Pierre > ========================================================================= > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple > xnatural.c:21 > [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple > xnatural.c:21 > [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > -------------------------------------------------------------------------- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -------------------------------------------------------------------------- > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > [0]PETSC ERROR: #1 PetscEventRegLogGetEvent() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/u > tils/eventlog.c:622 > [0]PETSC ERROR: #2 PetscLogEventRegister() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/plog > .c:802 > [0]PETSC ERROR: #3 VecInitializePackage() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface > /dlregisvec.c:187 > [0]PETSC ERROR: #4 VecCreate() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface/veccreate. > c:32 > [0]PETSC ERROR: #5 DMCreateLocalVector_Section_Private() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm > /interface/dmi.c:80 > [0]PETSC ERROR: #6 DMCreateLocalVector_Plex() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexcreate.c:4432 > [0]PETSC ERROR: #7 DMCreateLocalVector() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/interface/dm.c: > 1056 > [0]PETSC ERROR: #8 DMPlexCreateGmsh() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/plexgms > h.c:1933 > [0]PETSC ERROR: #9 DMPlexCreateGmshFromFile() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexgmsh.c:1433 > [0]PETSC ERROR: #10 JAF_DMPlexCreateFromMesh() at /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5845 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always exact. > [0]PETSC ERROR: #1 PetscStrcasecmp() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/utils/str.c:285 > [0]PETSC ERROR: #2 PetscEventRegLogGetEvent() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/u > tils/eventlog.c:622 > [0]PETSC ERROR: #3 PetscLogEventRegister() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/plog > .c:802 > [0]PETSC ERROR: #4 VecInitializePackage() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface > /dlregisvec.c:188 > [0]PETSC ERROR: #5 VecCreate() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface/veccreate. > c:32 > [0]PETSC ERROR: #6 DMCreateLocalVector_Section_Private() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm > /interface/dmi.c:80 > [0]PETSC ERROR: #7 DMCreateLocalVector_Plex() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexcreate.c:4432 > [0]PETSC ERROR: #8 DMCreateLocalVector() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/interface/dm.c: > 1056 > [0]PETSC ERROR: #9 DMPlexCreateGmsh() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/plexgms > h.c:1933 > [0]PETSC ERROR: #10 DMPlexCreateGmshFromFile() at /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/ple > x/plexgmsh.c:1433 > [0]PETSC ERROR: #11 JAF_DMPlexCreateFromMesh() at /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5845 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > -------------------------------------------------------------------------- > mpiexec detected that one or more processes exited with non-zero status, thus causing > the job to be terminated. The first process to do so was: > > Process name: [[33478,1],2] > Exit code: 15 > =========================================================================== > > Here are the options I give ./configure: > > ./configure --download-mpich=yes --download-viennacl=yes --download-hdf5=yes --download-chaco=yes --download-metis=yes --download-parmetis=yes --download-cgns=yes > From: Pierre Jolivet > > Sent: Monday, July 17, 2023 1:58 PM > To: Ferrand, Jesus A. > > Cc: petsc-users at mcs.anl.gov > > Subject: [EXTERNAL] Re: [petsc-users] PETSc Installation Assistance > > CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > https://petsc.org/release/faq/#what-does-the-message-hwloc-linux-ignoring-pci-device-with-non-16bit-domain-mean > > Thanks, > Pierre > >> On 17 Jul 2023, at 7:51 PM, Ferrand, Jesus A. > wrote: >> >> Greetings. >> >> I recently changed operating systems (Ubuntu 20.04 -> Debian 12 "Bookworm") and tried to reinstall PETSc. >> I tried doing the usual as described in (https://petsc.org/release/install/download/#recommended-obtain-release-version-with-git): >> git clone/pull >> ./configure -- ... -- >> make install >> make check >> Everything proceeds smoothly until the "make check" step, where I get the following error: >> ============================================================================== >> Running check examples to verify correct installation >> Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and PETSC_ARCH=arch-linux-c-debug >> Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> Number of SNES iterations = 2 >> Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> Number of SNES iterations = 2 >> 0a1,3 >> > hwloc/linux: Ignoring PCI device with non-16bit domain. >> > Pass --enable-32bits-pci-domain to configure to support such devices >> > (warning: it would break the library ABI, don't enable unless really needed). >> /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/tests >> Possible problem with ex47 running with hdf5, diffs above >> ========================================= >> Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> Number of SNES iterations = 3 >> Completed test examples >> Error while running make check >> gmake[1]: *** [makefile:123: check] Error 1 >> make: *** [GNUmakefile:17: check] Error 2 >> ============================================================================== >> >> >> I tried reinstalling the same version I was able to use prior to changing OS's (PETSc 3.18.3, via tarball) and get a similar error. >> ============================================================================== >> make PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_P >> arty/newPETSC/petsc-3.18.3 PETSC_ARCH=arch-linux-c-debug check >> Running check examples to verify correct installation >> Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3 and PETSC_ARCH=arch-linux-c-debug >> Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> Number of SNES iterations = 2 >> Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> Number of SNES iterations = 2 >> 0a1,3 >> > hwloc/linux: Ignoring PCI device with non-16bit domain. >> > Pass --enable-32bits-pci-domain to configure to support such devices >> > (warning: it would break the library ABI, don't enable unless really needed). >> /home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3/src/vec/vec/tests >> Possible problem with ex47 running with hdf5, diffs above >> ========================================= >> Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process >> See https://petsc.org/release/faq/ >> hwloc/linux: Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really needed). >> Number of SNES iterations = 3 >> Completed test examples >> Error while running make check >> gmake[1]: *** [makefile:149: check] Error 1 >> make: *** [GNUmakefile:17: check] Error 2 >> ============================================================================== >> >> >> I checked the FAQs (https://petsc.org/release/faq/#table-of-contents) and tried the suggestions there (specifically the points underhttps://petsc.org/release/faq/#what-does-it-mean-when-make-check-hangs-or-errors-on-petscoptionsinsertfile) to no avail. >> I don't use a VPN. >> I tried running make check with Wifi disabled. >> Ran ./configure with --download-mpich-device=ch3:nemesis --download-mpich >> >> Surprisingly, I can compile my PETSc programs using a makefile, and run them. >> However, the programs are now broken (I get SIGSEGV originating from within DM/DMPlex APIs). >> >> >> Sincerely: >> >> J.A. Ferrand >> Embry-Riddle Aeronautical University - Daytona Beach - FL >> Ph.D. Candidate, Aerospace Engineering >> M.Sc. Aerospace Engineering >> B.Sc. Aerospace Engineering >> B.Sc. Computational Mathematics >> >> Phone: (386)-843-1829 >> Email(s): ferranj2 at my.erau.edu >> jesus.ferrand at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 14:30:41 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 15:30:41 -0400 Subject: [petsc-users] Structured (DMDA) vs Unstructured (DMPlex) meshes In-Reply-To: <88F24FD0-4650-4B61-8FB4-C9B1EDB3FF9A@petsc.dev> References: <88F24FD0-4650-4B61-8FB4-C9B1EDB3FF9A@petsc.dev> Message-ID: On Mon, Jul 17, 2023 at 12:48?PM Barry Smith wrote: > > The largest potential advantage of DMDA is likely the possibility of > easily using geometric multigrid if it is appropriate for the problem (or > subproblem of the problem) you are solving. The second advantage is, this > depends on your PDE and discretization, the simplicity of your code, and > what part of PETSc it depends on, if it doesn't depend on DMPLEX (and > doesn't need to) then that is a good thing. Disadvantage of DMDA is lack > of flexibility in discretization, handling of non-standard boundary > conditions, geometry. > I will be more specific about the discretizations. DMDA supports collocated discretizations. If you have this, I would definitely recommend it, since the array programming style is easy. However, if you have mixed FEM, or higher order, or staggered discretizations, or complex BC, you need something else. There should be no difference in the solver since all the DMs just build Vec and Mat objects and hand them off. There is a potential difference for matrix-free operation. Right now, Plex is slow there, so we are putting in an interface to libCEED for that. Thanks, Matt > > On Jul 17, 2023, at 12:42 PM, Miguel Angel Salazar de Troya < > miguel.salazar at corintis.com> wrote: > > > > Hello, > > > > I am trying to understand if I should make the effort to make my code > use structured meshes instead of unstructured ones. My domain is cartesian > so that is the first check for structured meshes. However, the problem size > I am looking at is ~20 million degrees of freedom. My understanding is that > for this problem size, most of the time is spent on the solver. In this > case, do structured meshes still have an advantage? Can they run Krylov > methods faster than when using structured meshes? What about other solvers > and preconditioners? > > > > Thanks, > > Miguel > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 14:33:47 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 15:33:47 -0400 Subject: [petsc-users] Parallel matrix multiplication In-Reply-To: References: Message-ID: On Mon, Jul 17, 2023 at 2:22?PM Barry Smith wrote: > > > https://petsc.org/release/manualpages/Mat/MatPtAP/ also note that > PETSc has a large infrastructure for efficient ways to compute various > matrix-matrix operations with a variety of algorithms that can be all > accessed by starting with > https://petsc.org/release/manualpages/Mat/MatProductCreate/ > > > On Jul 17, 2023, at 1:13 PM, Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > I would like to perform the following operation > > [P^T A P] x? = P^T f > > If P is thin, you might want to consider making a MatLRC and passing that to a Krylov method, probably with a small precondition matrix (like P^T diag(A) P). Thanks, Matt > Where P is a rectangular matrix and A is a square matrix. All the > matrixes are constructed using MPIAIJ. > > Should I be concerned about the parallel partitioning of the matrix P and > A? > > Or can I just go ahead and use MatMatMult to compute [P^T A P]? > > Is there any in-built functionality to perform this operation? > > Kind regards, > Karthik. > > > > > > > > -- > *Dr. Karthik Chockalingam* > High-Performance Software Engineer > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 14:36:47 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 15:36:47 -0400 Subject: [petsc-users] [EXTERNAL] PETSc Installation Assistance In-Reply-To: References: <28151F30-4A69-4E37-9699-794F1ECB8F40@lip6.fr> Message-ID: Hi Jesus, I think you are on main. Did everything you have get rebuilt? Toby just rewrote large sections of logging and this is right where it fails for you. It should be easy to see what is wrong by running in the debugger. Thanks, Matt On Mon, Jul 17, 2023 at 3:11?PM Pierre Jolivet wrote: > > On 17 Jul 2023, at 9:00 PM, Ferrand, Jesus A. > wrote: > > Pierre: > Setting the environment variable allows make check to complete without > errors. > It only seems to run three checks. > I recall that in the past, make check would run 100+ tests. > > > You are probably thinking of make test, which is running thousands of > tests indeed. > > This is what I get from make check: > ========================================================================= > Running check examples to verify correct installation > Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and > PETSC_ARCH=arch-linux-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > > C/C++ example src/vec/vec/tests/ex47 run successfully with hdf5 > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > > Completed test examples > ========================================================================= > > Anyhow, some of the APIs that worked before now break. > For example "DMPlexCreateGmshFromFile()" breaks when I read simple test > mesh. > I tried configuring and installing both the newest release of PETSc and > that tarball of version 3.18.3. > Both seem to be breaking APIs that used to work in the older OS. > > > There has been some internal changes to the behavior of DMPlex, they > should be documented in the change log. > But you are saying that with the same version, just by switching OS, you > get a different behavior? > Do as Satish suggested then, because it?s unlikely that the root cause is > in the library itself. > > Thanks, > Pierre > > ========================================================================= > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and > https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always > exact. > [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple > xnatural.c:21 > [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at > /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 > message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and > https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always > exact. > [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple > xnatural.c:21 > [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at > /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 > message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > -------------------------------------------------------------------------- > > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -------------------------------------------------------------------------- > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and > https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always > exact. > [0]PETSC ERROR: #1 PetscEventRegLogGetEvent() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/u > tils/eventlog.c:622 > [0]PETSC ERROR: #2 PetscLogEventRegister() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/plog > .c:802 > [0]PETSC ERROR: #3 VecInitializePackage() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface > /dlregisvec.c:187 > [0]PETSC ERROR: #4 VecCreate() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface/veccreate. > c:32 > [0]PETSC ERROR: #5 DMCreateLocalVector_Section_Private() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm > /interface/dmi.c:80 > [0]PETSC ERROR: #6 DMCreateLocalVector_Plex() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexcreate.c:4432 > [0]PETSC ERROR: #7 DMCreateLocalVector() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/interface/dm.c: > 1056 > [0]PETSC ERROR: #8 DMPlexCreateGmsh() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/plexgms > h.c:1933 > [0]PETSC ERROR: #9 DMPlexCreateGmshFromFile() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexgmsh.c:1433 > [0]PETSC ERROR: #10 JAF_DMPlexCreateFromMesh() at > /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5845 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 > message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and > https://petsc.org/release/faq/ > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: The line numbers in the error traceback are not always > exact. > [0]PETSC ERROR: #1 PetscStrcasecmp() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/utils/str.c:285 > [0]PETSC ERROR: #2 PetscEventRegLogGetEvent() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/u > tils/eventlog.c:622 > [0]PETSC ERROR: #3 PetscLogEventRegister() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/sys/logging/plog > .c:802 > [0]PETSC ERROR: #4 VecInitializePackage() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface > /dlregisvec.c:188 > [0]PETSC ERROR: #5 VecCreate() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/interface/veccreate. > c:32 > [0]PETSC ERROR: #6 DMCreateLocalVector_Section_Private() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm > /interface/dmi.c:80 > [0]PETSC ERROR: #7 DMCreateLocalVector_Plex() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex > /plexcreate.c:4432 > [0]PETSC ERROR: #8 DMCreateLocalVector() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/interface/dm.c: > 1056 > [0]PETSC ERROR: #9 DMPlexCreateGmsh() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/plexgms > h.c:1933 > [0]PETSC ERROR: #10 DMPlexCreateGmshFromFile() at > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/ple > x/plexgmsh.c:1433 > [0]PETSC ERROR: #11 JAF_DMPlexCreateFromMesh() at > /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5845 > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59 > message=application called MPI_Abort(MPI_COMM_WORLD, 59) > - process 0 > : > system msg for write_line failure : Bad file descriptor > -------------------------------------------------------------------------- > > mpiexec detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[33478,1],2] > Exit code: 15 > =========================================================================== > > Here are the options I give ./configure: > > ./configure --download-mpich=yes --download-viennacl=yes > --download-hdf5=yes --download-chaco=yes --download-metis=yes > --download-parmetis=yes --download-cgns=yes > ------------------------------ > *From:* Pierre Jolivet > *Sent:* Monday, July 17, 2023 1:58 PM > *To:* Ferrand, Jesus A. > *Cc:* petsc-users at mcs.anl.gov > *Subject:* [EXTERNAL] Re: [petsc-users] PETSc Installation Assistance > > *CAUTION:* This email originated outside of Embry-Riddle Aeronautical > University. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > https://petsc.org/release/faq/#what-does-the-message-hwloc-linux-ignoring-pci-device-with-non-16bit-domain-mean > > Thanks, > Pierre > > On 17 Jul 2023, at 7:51 PM, Ferrand, Jesus A. > wrote: > > Greetings. > > I recently changed operating systems (Ubuntu 20.04 -> Debian 12 > "Bookworm") and tried to reinstall PETSc. > I tried doing the usual as described in ( > https://petsc.org/release/install/download/#recommended-obtain-release-version-with-git > ): > > - git clone/pull > - ./configure -- ... -- > - make install > - make check > > Everything proceeds smoothly until the "make check" step, where I get the > following error: > > ============================================================================== > Running check examples to verify correct installation > Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and > PETSC_ARCH=arch-linux-c-debug > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > 0a1,3 > > hwloc/linux: Ignoring PCI device with non-16bit domain. > > Pass --enable-32bits-pci-domain to configure to support such devices > > (warning: it would break the library ABI, don't enable unless really > needed). > /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/vec/vec/tests > Possible problem with ex47 running with hdf5, diffs above > ========================================= > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI > process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > Number of SNES iterations = 3 > Completed test examples > Error while running make check > gmake[1]: *** [makefile:123: check] Error 1 > make: *** [GNUmakefile:17: check] Error 2 > > ============================================================================== > > > I tried reinstalling the same version I was able to use prior to changing > OS's (PETSc 3.18.3, via tarball) and get a similar error. > > ============================================================================== > make PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_P > arty/newPETSC/petsc-3.18.3 PETSC_ARCH=arch-linux-c-debug check > Running check examples to verify correct installation > Using > PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3 and > PETSC_ARCH=arch-linux-c-debug > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > 0a1,3 > > hwloc/linux: Ignoring PCI device with non-16bit domain. > > Pass --enable-32bits-pci-domain to configure to support such devices > > (warning: it would break the library ABI, don't enable unless really > needed). > > /home/jesus/Desktop/JAF_NML/3rd_Party/newPETSC/petsc-3.18.3/src/vec/vec/tests > > Possible problem with ex47 running with hdf5, diffs above > ========================================= > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI > process > See https://petsc.org/release/faq/ > hwloc/linux: Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > Number of SNES iterations = 3 > Completed test examples > Error while running make check > gmake[1]: *** [makefile:149: check] Error 1 > make: *** [GNUmakefile:17: check] Error 2 > > ============================================================================== > > > I checked the FAQs (https://petsc.org/release/faq/#table-of-contents) and > tried the suggestions there (specifically the points under > https://petsc.org/release/faq/#what-does-it-mean-when-make-check-hangs-or-errors-on-petscoptionsinsertfile) > to no avail. > > - I don't use a VPN. > - I tried running make check with Wifi disabled. > - Ran ./configure with --download-mpich-device=ch3:nemesis > --download-mpich > > > Surprisingly, I can compile my PETSc programs using a makefile, and run > them. > However, the programs are now broken (I get SIGSEGV originating from > within DM/DMPlex APIs). > > > Sincerely: > *J.A. Ferrand* > Embry-Riddle Aeronautical University - Daytona Beach - FL > Ph.D. Candidate, Aerospace Engineering > M.Sc. Aerospace Engineering > B.Sc. Aerospace Engineering > B.Sc. Computational Mathematics > > *Phone:* (386)-843-1829 > *Email(s):* ferranj2 at my.erau.edu > jesus.ferrand at gmail.com > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 19:20:31 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 20:20:31 -0400 Subject: [petsc-users] Inquiry about reading the P2 tetrahedron mesh from GMSH In-Reply-To: References: Message-ID: On Fri, Jun 30, 2023 at 4:40?PM neil liu wrote: > Dear Petsc developers, > > I am reading P2 mesh from GMSH. And used DMFieldGetClosure_Internal to > check the coordinates for each tetrahedron, It seems reasonable. > But when I tried DMGetCoordinates (dm, &global), it seems the vector > global is not consistent with the node number, Then what is global here? > Reading high order coordinates works for quads/hexes, but not simplices right now because I need to figure out what order GMsh stores the coordinates in. Thus we are defaulting to affine coordinates here. Thanks, Matt > Thanks, > > Xiaodong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 17 19:21:34 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jul 2023 20:21:34 -0400 Subject: [petsc-users] Bug Report TaoALMM class In-Reply-To: References: Message-ID: Toby and Hansol, Has anyone looked at this? Thanks, Matt On Mon, Jun 12, 2023 at 8:24?AM Stephan K?hler < stephan.koehler at math.tu-freiberg.de> wrote: > Dear PETSc/Tao team, > > I think there might be a bug in the Tao ALMM class: In the function > TaoALMMComputeAugLagAndGradient_Private(), see, eg. > > https://petsc.org/release/src/tao/constrained/impls/almm/almm.c.html#TAOALMM > line 648 the gradient seems to be wrong. > > The given function and gradient computation is > Lc = F + Ye^TCe + Yi^T(Ci - S) + 0.5*mu*[Ce^TCe + (Ci - S)^T(Ci - S)], > dLc/dX = dF/dX + Ye^TAe + Yi^TAi + 0.5*mu*[Ce^TAe + (Ci - S)^TAi], > > but I think the gradient should be (without 0.5) > > dLc/dX = dF/dX + Ye^TAe + Yi^TAi + mu*[Ce^TAe + (Ci - S)^TAi]. > > Kind regards, > Stephan K?hler > > -- > Stephan K?hler > TU Bergakademie Freiberg > Institut f?r numerische Mathematik und Optimierung > > Akademiestra?e 6 > 09599 Freiberg > Geb?udeteil Mittelbau, Zimmer 2.07 > > Telefon: +49 (0)3731 39-3173 (B?ro) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From quentin.chevalier at polytechnique.edu Tue Jul 18 00:48:51 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Tue, 18 Jul 2023 07:48:51 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> Message-ID: Thank you for that pointer ! I have on hand a partial SVD of R, so I used that to build the approximate matrix instead. It's great that so many nice features of PETSc like STSetPreconditionerMat are accessible through petsc4py ! Good day, Quentin On Mon, 17 Jul 2023 at 17:29, Jose E. Roman wrote: > > It is possible to pass a different matrix to build the preconditioner. That is, the shell matrix for B (EPSSetOperators) and an explicit matrix (that approximates B) for the preconditioner. For instance, you can try passing M for building the preconditioner. Since M is an explicit matrix, you can try the default preconditioner (block Jacobi with ILU as local solver) or even a full LU decomposition. The effectiveness of the preconditioner will depend on how the update M+R^H P M P R moves the eigenvalues around. > > You can do this with STSetSplitPreconditioner() or STSetPreconditionerMat(). In your case any of them will do. > > Jose > > > > El 17 jul 2023, a las 15:50, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > > > > Thank you for this suggestion, I tried to implement that but it's > > proven pretty hard to implement MATOP_GET_DIAGONAL without completely > > tanking performance. After all, B is a shell matrix for a reason : it > > looks like M+R^H P M P R with R itself a shell matrix. > > > > Allow me to point out that I have no shift. My eigenvalue problem is > > purely about the largest ones out there. Section 8.2 and 3.4.3 led me > > to think that there was a way to avoid computing (or writing a shell > > matrix about it) B^-1... But you seem to stress that there's no way > > around it. > > > > Quentin > > > > > > > > On Mon, 17 Jul 2023 at 11:56, Jose E. Roman wrote: > >> > >> The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. > >> > >> If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. > >> > >> Jose > >> > >> > >>> El 17 jul 2023, a las 11:48, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > >>> > >>> Hello Jose, > >>> > >>> I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. > >>> > >>> I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. > >>> > >>> I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. > >>> > >>> Thank you for your time, > >>> > >>> Quentin > >>> > >>> > >>> On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > >>>> > >>>> By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > >>>> > >>>> An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > >>>> > >>>> Jose > >>>> > >>>> > >>>>> El 12 jul 2023, a las 19:04, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > >>>>> > >>>>> Hello PETSc Users, > >>>>> > >>>>> I have a generalised eigenvalue problem : Ax= lambda Bx > >>>>> I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > >>>>> > >>>>> Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > >>>>> > >>>>> I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > >>>>> > >>>>> A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > >>>>> > >>>>> I use PETSc in complex mode through the petsc4py bridge. > >>>>> > >>>>> Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > >>>>> > >>>>> Thank you for your time, > >>>>> > >>>>> Quentin > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From quentin.chevalier at polytechnique.edu Tue Jul 18 02:57:45 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Tue, 18 Jul 2023 09:57:45 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> Message-ID: Matrix to matrix products are taking much longer than expected... My snippet is below. m and n are quite large, >1M each. I'm running this on 35 procs. As you can see U, S and V are quite sparse SVD matrices (only their first 4 columns are dense, plus a chop). I expected therefore approximate R to have only rank 4 and computations to run smoothly once the cross products between U and V are computed... Right now my bottle neck is not preconditioner but getting that approximation of M2. What do you think ? def approximate(self,k:int): m, n = self.R.getSize() m_local,n_local = self.R.getLocalSize() I,J=self.tmp3.vector.getOwnershipRange() S=pet.Mat().create(comm=comm); S.setSizes([[n_local,n],[n_local,n]]) U=pet.Mat().create(comm=comm); U.setSizes([[m_local,m],[n_local,n]]) V=pet.Mat().create(comm=comm); V.setSizes([[n_local,n],[n_local,n]]) for A in (U,S,V): A.setUp() g=np.loadtxt("gains/txt/dummy.txt") for i in range(k): S.setValue(i,i,g[i]) loadStuff("left-vecs/", params,self.tmp3) U.setValues(self.reindex,[i],self.tmp3.x.array) loadStuff("right-vecs/",params,self.tmp3) V.setValues(range(I,J), [i],self.tmp3.vector) for A in (U,S,V): A.assemble() U.chop(1e-6); V.chop(1e-6) V.hermitianTranspose(V) self.P.matMult(U.matMult(S.matMult(V,S),U),U) # Multiplications in place (everyone is square except U) M2=self.N.duplicate() M2.matMult(U,U) U.hermitianTranspose().matMult(U,M2) return self.M+M2 Quentin On Tue, 18 Jul 2023 at 07:48, Quentin Chevalier < quentin.chevalier at polytechnique.edu> wrote: > > Thank you for that pointer ! I have on hand a partial SVD of R, so I used that to build the approximate matrix instead. > > It's great that so many nice features of PETSc like STSetPreconditionerMat are accessible through petsc4py ! > > Good day, > > Quentin > > > > On Mon, 17 Jul 2023 at 17:29, Jose E. Roman wrote: > > > > It is possible to pass a different matrix to build the preconditioner. That is, the shell matrix for B (EPSSetOperators) and an explicit matrix (that approximates B) for the preconditioner. For instance, you can try passing M for building the preconditioner. Since M is an explicit matrix, you can try the default preconditioner (block Jacobi with ILU as local solver) or even a full LU decomposition. The effectiveness of the preconditioner will depend on how the update M+R^H P M P R moves the eigenvalues around. > > > > You can do this with STSetSplitPreconditioner() or STSetPreconditionerMat(). In your case any of them will do. > > > > Jose > > > > > > > El 17 jul 2023, a las 15:50, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > > > > > > Thank you for this suggestion, I tried to implement that but it's > > > proven pretty hard to implement MATOP_GET_DIAGONAL without completely > > > tanking performance. After all, B is a shell matrix for a reason : it > > > looks like M+R^H P M P R with R itself a shell matrix. > > > > > > Allow me to point out that I have no shift. My eigenvalue problem is > > > purely about the largest ones out there. Section 8.2 and 3.4.3 led me > > > to think that there was a way to avoid computing (or writing a shell > > > matrix about it) B^-1... But you seem to stress that there's no way > > > around it. > > > > > > Quentin > > > > > > > > > > > > On Mon, 17 Jul 2023 at 11:56, Jose E. Roman wrote: > > >> > > >> The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. > > >> > > >> If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. > > >> > > >> Jose > > >> > > >> > > >>> El 17 jul 2023, a las 11:48, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > > >>> > > >>> Hello Jose, > > >>> > > >>> I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. > > >>> > > >>> I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. > > >>> > > >>> I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. > > >>> > > >>> Thank you for your time, > > >>> > > >>> Quentin > > >>> > > >>> > > >>> On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > > >>>> > > >>>> By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > > >>>> > > >>>> An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > > >>>> > > >>>> Jose > > >>>> > > >>>> > > >>>>> El 12 jul 2023, a las 19:04, Quentin Chevalier < quentin.chevalier at polytechnique.edu> escribi?: > > >>>>> > > >>>>> Hello PETSc Users, > > >>>>> > > >>>>> I have a generalised eigenvalue problem : Ax= lambda Bx > > >>>>> I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > >>>>> > > >>>>> Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > >>>>> > > >>>>> I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > >>>>> > > >>>>> A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > >>>>> > > >>>>> I use PETSc in complex mode through the petsc4py bridge. > > >>>>> > > >>>>> Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > >>>>> > > >>>>> Thank you for your time, > > >>>>> > > >>>>> Quentin > > >>>> > > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From quentin.chevalier at polytechnique.edu Tue Jul 18 03:04:03 2023 From: quentin.chevalier at polytechnique.edu (Quentin Chevalier) Date: Tue, 18 Jul 2023 10:04:03 +0200 Subject: [petsc-users] Matrix-free generalised eigenvalue problem In-Reply-To: References: <2BF04349-8CF1-4E68-87B7-290925A71114@dsic.upv.es> Message-ID: My apologies I didn't think the previous message through - the operation USV^H is far from 16 inner products and more like 1 M^2 inner products of length 4. I guess I should try to exploit sparsity of U and V (CSR works in parallel ?) but create a dense R. Cheers, Quentin Quentin CHEVALIER ? IA parcours recherche LadHyX - Ecole polytechnique __________ On Tue, 18 Jul 2023 at 09:57, Quentin Chevalier wrote: > > Matrix to matrix products are taking much longer than expected... My snippet is below. m and n are quite large, >1M each. I'm running this on 35 procs. As you can see U, S and V are quite sparse SVD matrices (only their first 4 columns are dense, plus a chop). I expected therefore approximate R to have only rank 4 and computations to run smoothly once the cross products between U and V are computed... Right now my bottle neck is not preconditioner but getting that approximation of M2. What do you think ? > > def approximate(self,k:int): > m, n = self.R.getSize() > m_local,n_local = self.R.getLocalSize() > I,J=self.tmp3.vector.getOwnershipRange() > S=pet.Mat().create(comm=comm); S.setSizes([[n_local,n],[n_local,n]]) > U=pet.Mat().create(comm=comm); U.setSizes([[m_local,m],[n_local,n]]) > V=pet.Mat().create(comm=comm); V.setSizes([[n_local,n],[n_local,n]]) > for A in (U,S,V): A.setUp() > g=np.loadtxt("gains/txt/dummy.txt") > for i in range(k): > S.setValue(i,i,g[i]) > loadStuff("left-vecs/", params,self.tmp3) > U.setValues(self.reindex,[i],self.tmp3.x.array) > loadStuff("right-vecs/",params,self.tmp3) > V.setValues(range(I,J), [i],self.tmp3.vector) > for A in (U,S,V): A.assemble() > U.chop(1e-6); V.chop(1e-6) > > V.hermitianTranspose(V) > self.P.matMult(U.matMult(S.matMult(V,S),U),U) # Multiplications in place (everyone is square except U) > M2=self.N.duplicate() > M2.matMult(U,U) > U.hermitianTranspose().matMult(U,M2) > return self.M+M2 > > Quentin > > > On Tue, 18 Jul 2023 at 07:48, Quentin Chevalier wrote: > > > > Thank you for that pointer ! I have on hand a partial SVD of R, so I used that to build the approximate matrix instead. > > > > It's great that so many nice features of PETSc like STSetPreconditionerMat are accessible through petsc4py ! > > > > Good day, > > > > Quentin > > > > > > > > On Mon, 17 Jul 2023 at 17:29, Jose E. Roman wrote: > > > > > > It is possible to pass a different matrix to build the preconditioner. That is, the shell matrix for B (EPSSetOperators) and an explicit matrix (that approximates B) for the preconditioner. For instance, you can try passing M for building the preconditioner. Since M is an explicit matrix, you can try the default preconditioner (block Jacobi with ILU as local solver) or even a full LU decomposition. The effectiveness of the preconditioner will depend on how the update M+R^H P M P R moves the eigenvalues around. > > > > > > You can do this with STSetSplitPreconditioner() or STSetPreconditionerMat(). In your case any of them will do. > > > > > > Jose > > > > > > > > > > El 17 jul 2023, a las 15:50, Quentin Chevalier escribi?: > > > > > > > > Thank you for this suggestion, I tried to implement that but it's > > > > proven pretty hard to implement MATOP_GET_DIAGONAL without completely > > > > tanking performance. After all, B is a shell matrix for a reason : it > > > > looks like M+R^H P M P R with R itself a shell matrix. > > > > > > > > Allow me to point out that I have no shift. My eigenvalue problem is > > > > purely about the largest ones out there. Section 8.2 and 3.4.3 led me > > > > to think that there was a way to avoid computing (or writing a shell > > > > matrix about it) B^-1... But you seem to stress that there's no way > > > > around it. > > > > > > > > Quentin > > > > > > > > > > > > > > > > On Mon, 17 Jul 2023 at 11:56, Jose E. Roman wrote: > > > >> > > > >> The B-inner product is independent of the ST operator. See Table 3.2. In generalized eigenproblems you always have an inverse. > > > >> > > > >> If your matrix is diagonally dominant, try implementing the MATOP_GET_DIAGONAL operation and using PCJACOBI. Apart from this, you have to build your own preconditioner. > > > >> > > > >> Jose > > > >> > > > >> > > > >>> El 17 jul 2023, a las 11:48, Quentin Chevalier escribi?: > > > >>> > > > >>> Hello Jose, > > > >>> > > > >>> I guess I expected B to not be inverted but instead used as a mass for a problem-specific inner product since I specified GHEP as a problem type. p50 of the same user manual seems to imply that that would indeed be the case. I don't see what problem there would be with using a shell B matrix as a weighting matrix, as long as a mat utility is provided of course. > > > >>> > > > >>> I tried the first approach - I set up my KSP as CG since B is hermitian positive-definite (I made a mistake in my first email), but I'm getting a KSPSolve has not converged, reason DIVERGED_ITS error. I'm letting it run for 1000 iterations already so it seems suspiciously slow for a CG solver. > > > >>> > > > >>> I'm grappling with a shell preconditioner now to try and speed it up, but I'm unsure which one allows for shell matrices. > > > >>> > > > >>> Thank you for your time, > > > >>> > > > >>> Quentin > > > >>> > > > >>> > > > >>> On Wed, 12 Jul 2023 at 19:24, Jose E. Roman wrote: > > > >>>> > > > >>>> By default, it is solving the problem as B^{-1}*A*x=lambda*x (see chapter on Spectral Transformation). That is why A can be a shell matrix without problem. But B needs to be an explicit matrix in order to compute an LU factorization. If B is also a shell matrix then you should set an iterative solver for the associated KSP (see examples in the chapter). > > > >>>> > > > >>>> An alternative is to create a shell matrix M that computes the action of B^{-1}*A, then pass M to the EPS solver as a standard eigenproblem. > > > >>>> > > > >>>> Jose > > > >>>> > > > >>>> > > > >>>>> El 12 jul 2023, a las 19:04, Quentin Chevalier escribi?: > > > >>>>> > > > >>>>> Hello PETSc Users, > > > >>>>> > > > >>>>> I have a generalised eigenvalue problem : Ax= lambda Bx > > > >>>>> I used to have only A as a matrix-free method, I used mumps and an LU preconditioner, everything worked fine. > > > >>>>> > > > >>>>> Now B is matrix-free as well, and my solver is returning an error : "MatSolverType mumps does not support matrix type python", which is ironic given it seem to handle A quite fine. > > > >>>>> > > > >>>>> I have read in the user manual here that there some methods may require additional methods to be supplied for B like MATOP_GET_DIAGONAL but it's unclear to me exactly what I should be implementing and what is the best solver for my case. > > > >>>>> > > > >>>>> A is hermitian, B is hermitian positive but not positive-definite or real. Therefore I have specified a GHEP problem type to the EPS object. > > > >>>>> > > > >>>>> I use PETSc in complex mode through the petsc4py bridge. > > > >>>>> > > > >>>>> Any help on how to get EPS to work for a generalised matrix-free case would be welcome. Performance is not a key issue here - I have a tractable high value case on hand. > > > >>>>> > > > >>>>> Thank you for your time, > > > >>>>> > > > >>>>> Quentin > > > >>>> > > > >> > > > From karthikeyan.chockalingam at stfc.ac.uk Tue Jul 18 08:00:51 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Tue, 18 Jul 2023 13:00:51 +0000 Subject: [petsc-users] periodic boundary conditions Message-ID: Hello, This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? Thank you for your help. Kind regards, Karthik. -- Dr. Karthik Chockalingam High Performance Software Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Tue Jul 18 08:31:02 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Jul 2023 09:31:02 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: Message-ID: On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > Hello, > > > > This is exactly not a PETSc question. I am solving a Poisson equation > using finite elements. I would like to impose PBC. I am thinking of using > the Lagrange multiplier method to impose them as constraints. Or do you > think I could take an alternative approach? > There are several options: 1) Just make a periodic mesh. This is what Plex does by default. 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could also do a Nitsche boundary condition for this. Since the constraint is so simple, I do not see an advantage to imposing it weakly. Thanks, Matt > Thank you for your help. > > > > Kind regards, > > Karthik. > > > > -- > > *Dr. Karthik Chockalingam* > > High Performance Software Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Tue Jul 18 10:18:10 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Tue, 18 Jul 2023 15:18:10 +0000 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: Message-ID: Thanks Matt. The mesh is structured (rectilinear), so it is periodic in that sense. Can you please explain how I can impose it strongly? My initial thought was to come up with a relation between the periodic nodes: x = P x? Say for 1-D problem with two elements (1)-------------(2)------------(3) P = [1 0, 0 1, 1 0] x = [x1 x2 x3] x? = [x1 x2] and solve [P^T A P] x? = P^T b I don?t think [P^T A P] is deterministic. Kind regards, Karthik. From: Matthew Knepley Date: Tuesday, 18 July 2023 at 14:31 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? There are several options: 1) Just make a periodic mesh. This is what Plex does by default. 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could also do a Nitsche boundary condition for this. Since the constraint is so simple, I do not see an advantage to imposing it weakly. Thanks, Matt Thank you for your help. Kind regards, Karthik. -- Dr. Karthik Chockalingam High Performance Software Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From knepley at gmail.com Tue Jul 18 10:38:17 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Jul 2023 11:38:17 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: Message-ID: On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Thanks Matt. > > > > The mesh is structured (rectilinear), so it is periodic in that sense. > > > > Can you please explain how I can impose it strongly? > Strongly means make those variables equal in a pointwise sense. We do this in the LocalToGlobal mapping, so one set is eliminated in the global problem. Thanks, Matt > My initial thought was to come up with a relation between the periodic > nodes: > > > > x = P x? > > > > Say for 1-D problem with two elements > > (1)-------------(2)------------(3) > > > > P = [1 0, 0 1, 1 0] > > x = [x1 x2 x3] > > x? = [x1 x2] > > > > and solve > > [P^T A P] x? = P^T b > > > > I don?t think [P^T A P] is deterministic. > > > > Kind regards, > > Karthik. > > > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 14:31 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > This is exactly not a PETSc question. I am solving a Poisson equation > using finite elements. I would like to impose PBC. I am thinking of using > the Lagrange multiplier method to impose them as constraints. Or do you > think I could take an alternative approach? > > > > There are several options: > > > > 1) Just make a periodic mesh. This is what Plex does by default. > > > > 2) Impose the conditions strongly. This is what is done if you create the > ZBox shape in Plex. > > > > 3) Impose the conditions weakly. This is what you are doing with Lagrange > multipliers. You could > > also do a Nitsche boundary condition for this. > > > > Since the constraint is so simple, I do not see an advantage to imposing > it weakly. > > > > Thanks, > > > > Matt > > > > Thank you for your help. > > > > Kind regards, > > Karthik. > > > > -- > > *Dr. Karthik Chockalingam* > > High Performance Software Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > [image: signature_3970890138] > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: not available URL: From karthikeyan.chockalingam at stfc.ac.uk Tue Jul 18 10:51:27 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Tue, 18 Jul 2023 15:51:27 +0000 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: Message-ID: Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? Best, Karthik. From: Matthew Knepley Date: Tuesday, 18 July 2023 at 16:38 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thanks Matt. The mesh is structured (rectilinear), so it is periodic in that sense. Can you please explain how I can impose it strongly? Strongly means make those variables equal in a pointwise sense. We do this in the LocalToGlobal mapping, so one set is eliminated in the global problem. Thanks, Matt My initial thought was to come up with a relation between the periodic nodes: x = P x? Say for 1-D problem with two elements (1)-------------(2)------------(3) P = [1 0, 0 1, 1 0] x = [x1 x2 x3] x? = [x1 x2] and solve [P^T A P] x? = P^T b I don?t think [P^T A P] is deterministic. Kind regards, Karthik. From: Matthew Knepley > Date: Tuesday, 18 July 2023 at 14:31 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? There are several options: 1) Just make a periodic mesh. This is what Plex does by default. 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could also do a Nitsche boundary condition for this. Since the constraint is so simple, I do not see an advantage to imposing it weakly. Thanks, Matt Thank you for your help. Kind regards, Karthik. -- Dr. Karthik Chockalingam High Performance Software Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk [signature_3970890138] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 18270 bytes Desc: image001.png URL: From bsmith at petsc.dev Tue Jul 18 10:58:09 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 18 Jul 2023 11:58:09 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: Message-ID: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> If you are using DMDA with periodic boundary conditions for example only one "copy" of such nodes exists in the global vector (the vector the solvers see) so one does not need to eliminate extra ones > On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: > > Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. > > Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? > > Best, > Karthik. > > > From: Matthew Knepley > > Date: Tuesday, 18 July 2023 at 16:38 > To: Chockalingam, Karthikeyan (STFC,DL,HC) > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: > Thanks Matt. > > The mesh is structured (rectilinear), so it is periodic in that sense. > > Can you please explain how I can impose it strongly? > > Strongly means make those variables equal in a pointwise sense. We do this in the > LocalToGlobal mapping, so one set is eliminated in the global problem. > > Thanks, > > Matt > > My initial thought was to come up with a relation between the periodic nodes: > > x = P x? > > Say for 1-D problem with two elements > (1)-------------(2)------------(3) > > P = [1 0, 0 1, 1 0] > x = [x1 x2 x3] > x? = [x1 x2] > > and solve > [P^T A P] x? = P^T b > > I don?t think [P^T A P] is deterministic. > > Kind regards, > Karthik. > > > From: Matthew Knepley > > Date: Tuesday, 18 July 2023 at 14:31 > To: Chockalingam, Karthikeyan (STFC,DL,HC) > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: > Hello, > > This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? > > There are several options: > > 1) Just make a periodic mesh. This is what Plex does by default. > > 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. > > 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could > also do a Nitsche boundary condition for this. > > Since the constraint is so simple, I do not see an advantage to imposing it weakly. > > Thanks, > > Matt > > Thank you for your help. > > Kind regards, > Karthik. > > -- > Dr. Karthik Chockalingam > High Performance Software Engineering Group > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Tue Jul 18 11:03:51 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Tue, 18 Jul 2023 16:03:51 +0000 Subject: [petsc-users] periodic boundary conditions In-Reply-To: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> Message-ID: Thank you, Barry. I am using the MPIAIJ format for a Finite Element application. So, I am trying to understand what is implemented in DMDA to eliminate those extra nodes. Best, Karthik. From: Barry Smith Date: Tuesday, 18 July 2023 at 16:58 To: Chockalingam, Karthikeyan (STFC,DL,HC) Cc: Matthew Knepley , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] periodic boundary conditions If you are using DMDA with periodic boundary conditions for example only one "copy" of such nodes exists in the global vector (the vector the solvers see) so one does not need to eliminate extra ones On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users wrote: Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? Best, Karthik. From: Matthew Knepley > Date: Tuesday, 18 July 2023 at 16:38 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thanks Matt. The mesh is structured (rectilinear), so it is periodic in that sense. Can you please explain how I can impose it strongly? Strongly means make those variables equal in a pointwise sense. We do this in the LocalToGlobal mapping, so one set is eliminated in the global problem. Thanks, Matt My initial thought was to come up with a relation between the periodic nodes: x = P x? Say for 1-D problem with two elements (1)-------------(2)------------(3) P = [1 0, 0 1, 1 0] x = [x1 x2 x3] x? = [x1 x2] and solve [P^T A P] x? = P^T b I don?t think [P^T A P] is deterministic. Kind regards, Karthik. From: Matthew Knepley > Date: Tuesday, 18 July 2023 at 14:31 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? There are several options: 1) Just make a periodic mesh. This is what Plex does by default. 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could also do a Nitsche boundary condition for this. Since the constraint is so simple, I do not see an advantage to imposing it weakly. Thanks, Matt Thank you for your help. Kind regards, Karthik. -- Dr. Karthik Chockalingam High Performance Software Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 18 11:07:17 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 18 Jul 2023 12:07:17 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> Message-ID: <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> They are never really "eliminated" because extra copies in the global vector never exist. > On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI wrote: > > Thank you, Barry. I am using the MPIAIJ format for a Finite Element application. So, I am trying to understand what is implemented in DMDA to eliminate those extra nodes. > > Best, > Karthik. > > From: Barry Smith > > Date: Tuesday, 18 July 2023 at 16:58 > To: Chockalingam, Karthikeyan (STFC,DL,HC) > > Cc: Matthew Knepley >, petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] periodic boundary conditions > > > If you are using DMDA with periodic boundary conditions for example only one "copy" of such nodes exists in the global vector (the vector the solvers see) so one does not need to eliminate extra ones > > > On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: > > Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. > > Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? > > Best, > Karthik. > > > From: Matthew Knepley > > Date: Tuesday, 18 July 2023 at 16:38 > To: Chockalingam, Karthikeyan (STFC,DL,HC) > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: > Thanks Matt. > > The mesh is structured (rectilinear), so it is periodic in that sense. > > Can you please explain how I can impose it strongly? > > Strongly means make those variables equal in a pointwise sense. We do this in the > LocalToGlobal mapping, so one set is eliminated in the global problem. > > Thanks, > > Matt > > My initial thought was to come up with a relation between the periodic nodes: > > x = P x? > > Say for 1-D problem with two elements > (1)-------------(2)------------(3) > > P = [1 0, 0 1, 1 0] > x = [x1 x2 x3] > x? = [x1 x2] > > and solve > [P^T A P] x? = P^T b > > I don?t think [P^T A P] is deterministic. > > Kind regards, > Karthik. > > > From: Matthew Knepley > > Date: Tuesday, 18 July 2023 at 14:31 > To: Chockalingam, Karthikeyan (STFC,DL,HC) > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: > Hello, > > This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? > > There are several options: > > 1) Just make a periodic mesh. This is what Plex does by default. > > 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. > > 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could > also do a Nitsche boundary condition for this. > > Since the constraint is so simple, I do not see an advantage to imposing it weakly. > > Thanks, > > Matt > > Thank you for your help. > > Kind regards, > Karthik. > > -- > Dr. Karthik Chockalingam > High Performance Software Engineering Group > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 18 11:22:34 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Jul 2023 12:22:34 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> Message-ID: Jed creates the LocalToGlobal that does this elimination in plexsfc.c Thanks, Matt On Tue, Jul 18, 2023 at 12:07?PM Barry Smith wrote: > > They are never really "eliminated" because extra copies in the global > vector never exist. > > On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thank you, Barry. I am using the MPIAIJ format for a Finite Element > application. So, I am trying to understand what is implemented in DMDA to > eliminate those extra nodes. > > Best, > Karthik. > > > *From: *Barry Smith > *Date: *Tuesday, 18 July 2023 at 16:58 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Matthew Knepley , petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] periodic boundary conditions > > If you are using DMDA with periodic boundary conditions for example > only one "copy" of such nodes exists in the global vector (the vector the > solvers see) so one does not need to eliminate extra ones > > > On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Yes, I clearly understand I need to eliminate one set of periodic nodes. I > was hoping to use x = P x? to eliminate one set. It is a kind of mapping. > > Sorry, I am not sure if it is the LocalToGlobal mapping you are referring > to. Is there an example or reference to show how the LocalToGlobal > mapping is being used to impose PBC? > > Best, > Karthik. > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 16:38 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] periodic boundary conditions > On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thanks Matt. > > The mesh is structured (rectilinear), so it is periodic in that sense. > > Can you please explain how I can impose it strongly? > > > Strongly means make those variables equal in a pointwise sense. We do this > in the > LocalToGlobal mapping, so one set is eliminated in the global problem. > > Thanks, > > Matt > > > My initial thought was to come up with a relation between the periodic > nodes: > > x = P x? > > Say for 1-D problem with two elements > (1)-------------(2)------------(3) > > P = [1 0, 0 1, 1 0] > x = [x1 x2 x3] > x? = [x1 x2] > > and solve > [P^T A P] x? = P^T b > > I don?t think [P^T A P] is deterministic. > > Kind regards, > Karthik. > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 14:31 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] periodic boundary conditions > On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > This is exactly not a PETSc question. I am solving a Poisson equation > using finite elements. I would like to impose PBC. I am thinking of using > the Lagrange multiplier method to impose them as constraints. Or do you > think I could take an alternative approach? > > > There are several options: > > 1) Just make a periodic mesh. This is what Plex does by default. > > 2) Impose the conditions strongly. This is what is done if you create the > ZBox shape in Plex. > > 3) Impose the conditions weakly. This is what you are doing with Lagrange > multipliers. You could > also do a Nitsche boundary condition for this. > > Since the constraint is so simple, I do not see an advantage to imposing > it weakly. > > Thanks, > > Matt > > > Thank you for your help. > > Kind regards, > Karthik. > > -- > *Dr. Karthik Chockalingam* > High Performance Software Engineering Group > Hartree Centre | Science and Technology Facilities Council > karthikeyan.chockalingam at stfc.ac.uk > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Wed Jul 19 05:23:19 2023 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Wed, 19 Jul 2023 10:23:19 +0000 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> Message-ID: I finally found the answer to my problem. I was not wrongly imposing the periodic boundary condition but rather was solving a problem (Poisson equation with PBC), which has an infinite family of solutions. I was recommended to add an artificial constraint or use a null-space-aware algebraic solver. Are MUMPS and Hypre null-space-aware algebraic solvers? How can I turn that option on? Kind regards, Karthik. From: Matthew Knepley Date: Tuesday, 18 July 2023 at 17:22 To: Barry Smith Cc: Chockalingam, Karthikeyan (STFC,DL,HC) , petsc-users at mcs.anl.gov Subject: Re: [petsc-users] periodic boundary conditions Jed creates the LocalToGlobal that does this elimination in plexsfc.c Thanks, Matt On Tue, Jul 18, 2023 at 12:07?PM Barry Smith > wrote: They are never really "eliminated" because extra copies in the global vector never exist. On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI > wrote: Thank you, Barry. I am using the MPIAIJ format for a Finite Element application. So, I am trying to understand what is implemented in DMDA to eliminate those extra nodes. Best, Karthik. From: Barry Smith > Date: Tuesday, 18 July 2023 at 16:58 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: Matthew Knepley >, petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions If you are using DMDA with periodic boundary conditions for example only one "copy" of such nodes exists in the global vector (the vector the solvers see) so one does not need to eliminate extra ones On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? Best, Karthik. From: Matthew Knepley > Date: Tuesday, 18 July 2023 at 16:38 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thanks Matt. The mesh is structured (rectilinear), so it is periodic in that sense. Can you please explain how I can impose it strongly? Strongly means make those variables equal in a pointwise sense. We do this in the LocalToGlobal mapping, so one set is eliminated in the global problem. Thanks, Matt My initial thought was to come up with a relation between the periodic nodes: x = P x? Say for 1-D problem with two elements (1)-------------(2)------------(3) P = [1 0, 0 1, 1 0] x = [x1 x2 x3] x? = [x1 x2] and solve [P^T A P] x? = P^T b I don?t think [P^T A P] is deterministic. Kind regards, Karthik. From: Matthew Knepley > Date: Tuesday, 18 July 2023 at 14:31 To: Chockalingam, Karthikeyan (STFC,DL,HC) > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] periodic boundary conditions On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: Hello, This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? There are several options: 1) Just make a periodic mesh. This is what Plex does by default. 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could also do a Nitsche boundary condition for this. Since the constraint is so simple, I do not see an advantage to imposing it weakly. Thanks, Matt Thank you for your help. Kind regards, Karthik. -- Dr. Karthik Chockalingam High Performance Software Engineering Group Hartree Centre | Science and Technology Facilities Council karthikeyan.chockalingam at stfc.ac.uk -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 19 05:26:15 2023 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 19 Jul 2023 06:26:15 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> Message-ID: On Wed, Jul 19, 2023 at 6:23?AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > I finally found the answer to my problem. I was not wrongly imposing the > periodic boundary condition but rather was solving a problem (Poisson > equation with PBC), which has an infinite family of solutions. I was > recommended to add an artificial constraint or use a null-space-aware > algebraic solver. > > > > Are MUMPS and Hypre null-space-aware algebraic solvers? How can I turn > that option on? > MUMPS can do it with an option (it is in the documentation). For AMG you need to specify the coarse solver. I know this works with GAMG, you use -mg_coarse_pc_type svd Thanks, Matt > Kind regards, > > Karthik. > > > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 17:22 > *To: *Barry Smith > *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk>, petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] periodic boundary conditions > > Jed creates the LocalToGlobal that does this elimination in plexsfc.c > > > > Thanks, > > > > Matt > > > > On Tue, Jul 18, 2023 at 12:07?PM Barry Smith wrote: > > > > They are never really "eliminated" because extra copies in the global > vector never exist. > > > > On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > > > Thank you, Barry. I am using the MPIAIJ format for a Finite Element > application. So, I am trying to understand what is implemented in DMDA to > eliminate those extra nodes. > > > > Best, > > Karthik. > > > > *From: *Barry Smith > *Date: *Tuesday, 18 July 2023 at 16:58 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Matthew Knepley , petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] periodic boundary conditions > > > > If you are using DMDA with periodic boundary conditions for example > only one "copy" of such nodes exists in the global vector (the vector the > solvers see) so one does not need to eliminate extra ones > > > > On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > > > Yes, I clearly understand I need to eliminate one set of periodic nodes. I > was hoping to use x = P x? to eliminate one set. It is a kind of mapping. > > > > Sorry, I am not sure if it is the LocalToGlobal mapping you are referring > to. Is there an example or reference to show how the LocalToGlobal mapping > is being used to impose PBC? > > > > Best, > > Karthik. > > > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 16:38 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thanks Matt. > > > > The mesh is structured (rectilinear), so it is periodic in that sense. > > > > Can you please explain how I can impose it strongly? > > > > Strongly means make those variables equal in a pointwise sense. We do this > in the > > LocalToGlobal mapping, so one set is eliminated in the global problem. > > > > Thanks, > > > > Matt > > > > My initial thought was to come up with a relation between the periodic > nodes: > > > > x = P x? > > > > Say for 1-D problem with two elements > > (1)-------------(2)------------(3) > > > > P = [1 0, 0 1, 1 0] > > x = [x1 x2 x3] > > x? = [x1 x2] > > > > and solve > > [P^T A P] x? = P^T b > > > > I don?t think [P^T A P] is deterministic. > > > > Kind regards, > > Karthik. > > > > > > *From: *Matthew Knepley > *Date: *Tuesday, 18 July 2023 at 14:31 > *To: *Chockalingam, Karthikeyan (STFC,DL,HC) < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *petsc-users at mcs.anl.gov > *Subject: *Re: [petsc-users] periodic boundary conditions > > On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via > petsc-users wrote: > > Hello, > > > > This is exactly not a PETSc question. I am solving a Poisson equation > using finite elements. I would like to impose PBC. I am thinking of using > the Lagrange multiplier method to impose them as constraints. Or do you > think I could take an alternative approach? > > > > There are several options: > > > > 1) Just make a periodic mesh. This is what Plex does by default. > > > > 2) Impose the conditions strongly. This is what is done if you create the > ZBox shape in Plex. > > > > 3) Impose the conditions weakly. This is what you are doing with Lagrange > multipliers. You could > > also do a Nitsche boundary condition for this. > > > > Since the constraint is so simple, I do not see an advantage to imposing > it weakly. > > > > Thanks, > > > > Matt > > > > Thank you for your help. > > > > Kind regards, > > Karthik. > > > > -- > > *Dr. Karthik Chockalingam* > > High Performance Software Engineering Group > > Hartree Centre | Science and Technology Facilities Council > > karthikeyan.chockalingam at stfc.ac.uk > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karsten at bolding-bruggeman.com Wed Jul 19 06:58:31 2023 From: karsten at bolding-bruggeman.com (Karsten Bolding) Date: Wed, 19 Jul 2023 13:58:31 +0200 Subject: [petsc-users] Python version of VecDuplicateVecs Message-ID: Hello I'm re-implementing a model from mixed C/Fortran to Python where Petsc is used for matrix/vector operations. In the C-code I have: ierr = VecDuplicateVecs(templateVec,numTracers,&uef); to create uef with numTracers vectors. Later in the code looping is done of the individual vectors and a sequence of operations are performed. I would like to follow the same idea in Python - but do not find any example. An alternative would be to make a list of Vecs? Best Regards, Karsten -- Karsten Bolding karsten at bolding-bruggeman.com +4564422058 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 19 07:25:39 2023 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 19 Jul 2023 08:25:39 -0400 Subject: [petsc-users] Python version of VecDuplicateVecs In-Reply-To: References: Message-ID: On Wed, Jul 19, 2023 at 7:59?AM Karsten Bolding < karsten at bolding-bruggeman.com> wrote: > Hello > > I'm re-implementing a model from mixed C/Fortran to Python where Petsc is > used for matrix/vector operations. > > In the C-code I have: > ierr = VecDuplicateVecs(templateVec,numTracers,&uef); > to create uef with numTracers vectors. > > Later in the code looping is done of the individual vectors and a sequence > of operations are performed. > > I would like to follow the same idea in Python - but do not find any > example. > > An alternative would be to make a list of Vecs? > There is no optimization here, so I would just make a list of duplicates. Thanks, Matt > Best Regards, > Karsten > > -- > Karsten Bolding > karsten at bolding-bruggeman.com > +4564422058 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 19 09:32:34 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 19 Jul 2023 10:32:34 -0400 Subject: [petsc-users] periodic boundary conditions In-Reply-To: References: <61B66B0D-708C-4CE1-A02C-0FF19F3B8A79@petsc.dev> <97E3E016-A2BE-4152-81F0-F64DAA53B0D6@petsc.dev> Message-ID: <9DB3E739-2560-405C-A3F0-7A1D37ABE6DB@petsc.dev> You may also need https://petsc.org/release/manualpages/Mat/MatNullSpaceCreate/#matnullspacecreate and https://petsc.org/release/manualpages/Mat/MatSetNullSpace/ > On Jul 19, 2023, at 6:26 AM, Matthew Knepley wrote: > > On Wed, Jul 19, 2023 at 6:23?AM Karthikeyan Chockalingam - STFC UKRI > wrote: >> I finally found the answer to my problem. I was not wrongly imposing the periodic boundary condition but rather was solving a problem (Poisson equation with PBC), which has an infinite family of solutions. I was recommended to add an artificial constraint or use a null-space-aware algebraic solver. >> >> >> >> Are MUMPS and Hypre null-space-aware algebraic solvers? How can I turn that option on? >> > > MUMPS can do it with an option (it is in the documentation). For AMG you need to specify the coarse solver. I know this works with GAMG, you use > > -mg_coarse_pc_type svd > > Thanks, > > Matt > >> Kind regards, >> >> Karthik. >> >> >> >> >> >> From: Matthew Knepley > >> Date: Tuesday, 18 July 2023 at 17:22 >> To: Barry Smith > >> Cc: Chockalingam, Karthikeyan (STFC,DL,HC) >, petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] periodic boundary conditions >> >> Jed creates the LocalToGlobal that does this elimination in plexsfc.c >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> On Tue, Jul 18, 2023 at 12:07?PM Barry Smith > wrote: >> >> >> >> They are never really "eliminated" because extra copies in the global vector never exist. >> >> >> >> >> On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI > wrote: >> >> >> >> Thank you, Barry. I am using the MPIAIJ format for a Finite Element application. So, I am trying to understand what is implemented in DMDA to eliminate those extra nodes. >> >> >> >> Best, >> >> Karthik. >> >> >> >> From: Barry Smith > >> Date: Tuesday, 18 July 2023 at 16:58 >> To: Chockalingam, Karthikeyan (STFC,DL,HC) > >> Cc: Matthew Knepley >, petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] periodic boundary conditions >> >> >> >> If you are using DMDA with periodic boundary conditions for example only one "copy" of such nodes exists in the global vector (the vector the solvers see) so one does not need to eliminate extra ones >> >> >> >> On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: >> >> >> >> Yes, I clearly understand I need to eliminate one set of periodic nodes. I was hoping to use x = P x? to eliminate one set. It is a kind of mapping. >> >> >> >> Sorry, I am not sure if it is the LocalToGlobal mapping you are referring to. Is there an example or reference to show how the LocalToGlobal mapping is being used to impose PBC? >> >> >> >> Best, >> >> Karthik. >> >> >> >> >> >> From: Matthew Knepley > >> Date: Tuesday, 18 July 2023 at 16:38 >> To: Chockalingam, Karthikeyan (STFC,DL,HC) > >> Cc: petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] periodic boundary conditions >> >> On Tue, Jul 18, 2023 at 11:18?AM Karthikeyan Chockalingam - STFC UKRI > wrote: >> >> Thanks Matt. >> >> >> >> The mesh is structured (rectilinear), so it is periodic in that sense. >> >> >> >> Can you please explain how I can impose it strongly? >> >> >> >> Strongly means make those variables equal in a pointwise sense. We do this in the >> >> LocalToGlobal mapping, so one set is eliminated in the global problem. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> My initial thought was to come up with a relation between the periodic nodes: >> >> >> >> x = P x? >> >> >> >> Say for 1-D problem with two elements >> >> (1)-------------(2)------------(3) >> >> >> >> P = [1 0, 0 1, 1 0] >> >> x = [x1 x2 x3] >> >> x? = [x1 x2] >> >> >> >> and solve >> >> [P^T A P] x? = P^T b >> >> >> >> I don?t think [P^T A P] is deterministic. >> >> >> >> Kind regards, >> >> Karthik. >> >> >> >> >> >> From: Matthew Knepley > >> Date: Tuesday, 18 July 2023 at 14:31 >> To: Chockalingam, Karthikeyan (STFC,DL,HC) > >> Cc: petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] periodic boundary conditions >> >> On Tue, Jul 18, 2023 at 9:02?AM Karthikeyan Chockalingam - STFC UKRI via petsc-users > wrote: >> >> Hello, >> >> >> >> This is exactly not a PETSc question. I am solving a Poisson equation using finite elements. I would like to impose PBC. I am thinking of using the Lagrange multiplier method to impose them as constraints. Or do you think I could take an alternative approach? >> >> >> >> There are several options: >> >> >> >> 1) Just make a periodic mesh. This is what Plex does by default. >> >> >> >> 2) Impose the conditions strongly. This is what is done if you create the ZBox shape in Plex. >> >> >> >> 3) Impose the conditions weakly. This is what you are doing with Lagrange multipliers. You could >> >> also do a Nitsche boundary condition for this. >> >> >> >> Since the constraint is so simple, I do not see an advantage to imposing it weakly. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> Thank you for your help. >> >> >> >> Kind regards, >> >> Karthik. >> >> >> >> -- >> >> Dr. Karthik Chockalingam >> >> High Performance Software Engineering Group >> >> Hartree Centre | Science and Technology Facilities Council >> >> karthikeyan.chockalingam at stfc.ac.uk >> >> >> >> >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.stone at opengosim.com Wed Jul 19 09:52:51 2023 From: daniel.stone at opengosim.com (Daniel Stone) Date: Wed, 19 Jul 2023 15:52:51 +0100 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre Message-ID: Hello, I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works by setting up a VS project to run some of the example programs. My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: #include "confdefs.h" #include "conffix.h" /* Override any gcc2 internal prototype to avoid an error. */ #include "HYPRE.h" char HYPRE_IJMatrixCreate(); static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } int main() { _check_HYPRE_IJMatrixCreate();; return 0; } As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that include PTScotch. I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more nontrivial Hypre example programs, I get the same error: Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? Many thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 19 10:15:29 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 19 Jul 2023 11:15:29 -0400 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: References: Message-ID: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. Barry The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > On Jul 19, 2023, at 10:52 AM, Daniel Stone wrote: > > Hello, > > I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > by setting up a VS project to run some of the example programs. > > My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > > I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > > #include "confdefs.h" > #include "conffix.h" > /* Override any gcc2 internal prototype to avoid an error. */ > > #include "HYPRE.h" > > char HYPRE_IJMatrixCreate(); > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > int main() { > _check_HYPRE_IJMatrixCreate();; > return 0; > } > > > As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > include PTScotch. > > I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > > unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > > Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > nontrivial Hypre example programs, I get the same error: > > Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > > Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > > Many thanks, > > Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 19 10:19:43 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 19 Jul 2023 11:19:43 -0400 Subject: [petsc-users] PETSc Discord server In-Reply-To: <77BA1846-5127-45CD-9C3A-26DAA5B052BF@petsc.dev> References: <77BA1846-5127-45CD-9C3A-26DAA5B052BF@petsc.dev> Message-ID: Here is an update address to join the PETSc Discord space https://discord.gg/Fqm8r6Gcyb > On Jul 5, 2023, at 10:04 PM, Barry Smith wrote: > > > We are experimenting with using Discord as our PETSc chat space. PETSc users are welcome to join us there https://discord.gg/TAmq3kvB > > Barry > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jul 19 10:30:12 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 Jul 2023 10:30:12 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> Message-ID: <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py index 5bc88322aa2..2d6c7932e17 100644 --- a/config/BuildSystem/config/packages/hypre.py +++ b/config/BuildSystem/config/packages/hypre.py @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): self.requiresversion = 1 self.gitcommit = 'v'+self.version self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] - self.functions = ['HYPRE_IJMatrixCreate'] + self.functions = [] self.includes = ['HYPRE.h'] self.liblist = [['libHYPRE.a']] self.buildLanguages = ['C','Cxx'] Satish On Wed, 19 Jul 2023, Barry Smith wrote: > > You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared > > I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. > > The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. > > Barry > > The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > > > > > > > > > On Jul 19, 2023, at 10:52 AM, Daniel Stone wrote: > > > > Hello, > > > > I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > > by setting up a VS project to run some of the example programs. > > > > My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > > to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > > through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > > > > I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > > > > #include "confdefs.h" > > #include "conffix.h" > > /* Override any gcc2 internal prototype to avoid an error. */ > > > > #include "HYPRE.h" > > > > char HYPRE_IJMatrixCreate(); > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > int main() { > > _check_HYPRE_IJMatrixCreate();; > > return 0; > > } > > > > > > As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > > include PTScotch. > > > > I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > > > > unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > > > > Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > > nontrivial Hypre example programs, I get the same error: > > > > Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > > > So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > > it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > > > > Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > > > > Many thanks, > > > > Daniel > > From balay at mcs.anl.gov Wed Jul 19 10:40:04 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 Jul 2023 10:40:04 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> Message-ID: BTW: Some explanation of configure: It attempts the following on linux: >>>>>> Source: #include "confdefs.h" #include "conffix.h" /* Override any gcc2 internal prototype to avoid an error. */ char HYPRE_IJMatrixCreate(); static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } int main(void) { _check_HYPRE_IJMatrixCreate(); return 0; } <<<<<<< Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); Compiling it manually: >>>> [balay at pj01 petsc]$ cat conftest.c char HYPRE_IJMatrixCreate(); static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } int main(void) { _check_HYPRE_IJMatrixCreate(); return 0; } [balay at pj01 petsc]$ gcc -c conftest.c [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate conftest.o: U HYPRE_IJMatrixCreate [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate [balay at pj01 petsc]$ <<<< Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! >>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so [balay at pj01 petsc]$ echo $? 0 <<<<< On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. Satish On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. > > diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py > index 5bc88322aa2..2d6c7932e17 100644 > --- a/config/BuildSystem/config/packages/hypre.py > +++ b/config/BuildSystem/config/packages/hypre.py > @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > self.requiresversion = 1 > self.gitcommit = 'v'+self.version > self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > - self.functions = ['HYPRE_IJMatrixCreate'] > + self.functions = [] > self.includes = ['HYPRE.h'] > self.liblist = [['libHYPRE.a']] > self.buildLanguages = ['C','Cxx'] > > > Satish > > > On Wed, 19 Jul 2023, Barry Smith wrote: > > > > > You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared > > > > I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. > > > > The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. > > > > Barry > > > > The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > > > > > > > > > > > > > > > > > On Jul 19, 2023, at 10:52 AM, Daniel Stone wrote: > > > > > > Hello, > > > > > > I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > > > by setting up a VS project to run some of the example programs. > > > > > > My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > > > to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > > > through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > > > > > > I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > > > > > > #include "confdefs.h" > > > #include "conffix.h" > > > /* Override any gcc2 internal prototype to avoid an error. */ > > > > > > #include "HYPRE.h" > > > > > > char HYPRE_IJMatrixCreate(); > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > int main() { > > > _check_HYPRE_IJMatrixCreate();; > > > return 0; > > > } > > > > > > > > > As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > > > include PTScotch. > > > > > > I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > > > > > > unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > > > > > > Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > > > nontrivial Hypre example programs, I get the same error: > > > > > > Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > > > > > So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > > > it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > > > > > > Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > > > > > > Many thanks, > > > > > > Daniel > > > > > From bsmith at petsc.dev Wed Jul 19 10:45:37 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 19 Jul 2023 11:45:37 -0400 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> Message-ID: <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> Satish, So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). Barry > On Jul 19, 2023, at 11:40 AM, Satish Balay wrote: > > BTW: Some explanation of configure: > > It attempts the following on linux: > >>>>>>> > Source: > #include "confdefs.h" > #include "conffix.h" > /* Override any gcc2 internal prototype to avoid an error. */ > char HYPRE_IJMatrixCreate(); > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > int main(void) { > _check_HYPRE_IJMatrixCreate(); > return 0; > } > <<<<<<< > > Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); > > Compiling it manually: > >>>>> > [balay at pj01 petsc]$ cat conftest.c > char HYPRE_IJMatrixCreate(); > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > int main(void) { > _check_HYPRE_IJMatrixCreate(); > return 0; > } > [balay at pj01 petsc]$ gcc -c conftest.c > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > conftest.o: U HYPRE_IJMatrixCreate > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate > [balay at pj01 petsc]$ > <<<< > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! > >>>>>>> > [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so > [balay at pj01 petsc]$ echo $? > 0 > <<<<< > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. > > Satish > > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > >> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. >> >> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py >> index 5bc88322aa2..2d6c7932e17 100644 >> --- a/config/BuildSystem/config/packages/hypre.py >> +++ b/config/BuildSystem/config/packages/hypre.py >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): >> self.requiresversion = 1 >> self.gitcommit = 'v'+self.version >> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] >> - self.functions = ['HYPRE_IJMatrixCreate'] >> + self.functions = [] >> self.includes = ['HYPRE.h'] >> self.liblist = [['libHYPRE.a']] >> self.buildLanguages = ['C','Cxx'] >> >> >> Satish >> >> >> On Wed, 19 Jul 2023, Barry Smith wrote: >> >>> >>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared >>> >>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. >>> >>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. >>> >>> Barry >>> >>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. >>> >>> >>> >>> >>> >>> >>> >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone wrote: >>>> >>>> Hello, >>>> >>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works >>>> by setting up a VS project to run some of the example programs. >>>> >>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications >>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). >>>> >>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: >>>> >>>> #include "confdefs.h" >>>> #include "conffix.h" >>>> /* Override any gcc2 internal prototype to avoid an error. */ >>>> >>>> #include "HYPRE.h" >>>> >>>> char HYPRE_IJMatrixCreate(); >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>> >>>> int main() { >>>> _check_HYPRE_IJMatrixCreate();; >>>> return 0; >>>> } >>>> >>>> >>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that >>>> include PTScotch. >>>> >>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: >>>> >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... >>>> >>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more >>>> nontrivial Hypre example programs, I get the same error: >>>> >>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 >>>> >>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but >>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. >>>> >>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? >>>> >>>> Many thanks, >>>> >>>> Daniel >>> >>> >> > From balay at mcs.anl.gov Wed Jul 19 10:58:47 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 Jul 2023 10:58:47 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> Message-ID: <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> I think it should work with static libraries and 64bit compilers. That's how I think --download-f2cblaslapack [etc] work. Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 $ nm -Ao msmpi.lib |grep " MPI_Init" msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. Checking PETSc dll build: balay at ps5 ~/petsc/arch-ci-mswin-uni/lib $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays It also has the unmangled symbol - so I guess this mode can work generally with dlls. Satish On Wed, 19 Jul 2023, Barry Smith wrote: > > Satish, > > So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). > > Barry > > > > On Jul 19, 2023, at 11:40 AM, Satish Balay wrote: > > > > BTW: Some explanation of configure: > > > > It attempts the following on linux: > > > >>>>>>> > > Source: > > #include "confdefs.h" > > #include "conffix.h" > > /* Override any gcc2 internal prototype to avoid an error. */ > > char HYPRE_IJMatrixCreate(); > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > int main(void) { > > _check_HYPRE_IJMatrixCreate(); > > return 0; > > } > > <<<<<<< > > > > Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); > > > > Compiling it manually: > > > >>>>> > > [balay at pj01 petsc]$ cat conftest.c > > char HYPRE_IJMatrixCreate(); > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > int main(void) { > > _check_HYPRE_IJMatrixCreate(); > > return 0; > > } > > [balay at pj01 petsc]$ gcc -c conftest.c > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > > conftest.o: U HYPRE_IJMatrixCreate > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate > > [balay at pj01 petsc]$ > > <<<< > > > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! > > > >>>>>>> > > [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so > > [balay at pj01 petsc]$ echo $? > > 0 > > <<<<< > > > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. > > > > Satish > > > > > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > > > >> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. > >> > >> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py > >> index 5bc88322aa2..2d6c7932e17 100644 > >> --- a/config/BuildSystem/config/packages/hypre.py > >> +++ b/config/BuildSystem/config/packages/hypre.py > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > >> self.requiresversion = 1 > >> self.gitcommit = 'v'+self.version > >> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > >> - self.functions = ['HYPRE_IJMatrixCreate'] > >> + self.functions = [] > >> self.includes = ['HYPRE.h'] > >> self.liblist = [['libHYPRE.a']] > >> self.buildLanguages = ['C','Cxx'] > >> > >> > >> Satish > >> > >> > >> On Wed, 19 Jul 2023, Barry Smith wrote: > >> > >>> > >>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared > >>> > >>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. > >>> > >>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. > >>> > >>> Barry > >>> > >>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone wrote: > >>>> > >>>> Hello, > >>>> > >>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > >>>> by setting up a VS project to run some of the example programs. > >>>> > >>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > >>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > >>>> > >>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > >>>> > >>>> #include "confdefs.h" > >>>> #include "conffix.h" > >>>> /* Override any gcc2 internal prototype to avoid an error. */ > >>>> > >>>> #include "HYPRE.h" > >>>> > >>>> char HYPRE_IJMatrixCreate(); > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >>>> > >>>> int main() { > >>>> _check_HYPRE_IJMatrixCreate();; > >>>> return 0; > >>>> } > >>>> > >>>> > >>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > >>>> include PTScotch. > >>>> > >>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > >>>> > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > >>>> > >>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > >>>> nontrivial Hypre example programs, I get the same error: > >>>> > >>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > >>>> > >>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > >>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > >>>> > >>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > >>>> > >>>> Many thanks, > >>>> > >>>> Daniel > >>> > >>> > >> > > > From Robert.Crockett at lamresearch.com Wed Jul 19 15:31:31 2023 From: Robert.Crockett at lamresearch.com (Robert.Crockett at lamresearch.com) Date: Wed, 19 Jul 2023 20:31:31 +0000 Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC Message-ID: Hello, I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. See the attached for more information. * The file petscbuild.sh is the script used to configure and build * The file configure.log is output by PETSc * The file log.petscbuild is the output of the config & build script. Can you please help find a work-around? Best, Robert PS. Some related links I found in searching on this issue. The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. https://community.intel.com/t5/Intel-C-Compiler/Default-constructor-of-variant-is-deleted/m-p/1156212?profile.language=en https://mediatum.ub.tum.de/doc/1555265/1555265.pdf Robert Crockett Plasma Simulation Engineer | OCTO - Computational Products P: 617.648.8349 M: 415.205.4567 LAM RESEARCH 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com [cid:image001.png at 01D9BA5D.0F372F60] ----------------------------------------- Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic mpicc -show: icc -fPIC -wd1572 -O3 -mtune=generic -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi C compiler version: icc (ICC) 18.0.5 20180823 Using C++ compile: /usr/local/petsc/r/bin/mpicxx -o .o -c -wd1572 -O3 -mtune=generic -std=c++14 -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include mpicxx -show: icpc -wd1572 -O3 -mtune=generic -std=c++14 -fPIC -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi C++ compiler version: icpc (ICC) 18.0.5 20180823 Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include mpif90 -show: gfortran -fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune=generic -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi Fortran compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) ----------------------------------------- Using C/C++ linker: /usr/local/petsc/r/bin/mpicc Using C/C++ flags: -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic Using Fortran linker: /usr/local/petsc/r/bin/mpif90 Using Fortran flags: -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic ----------------------------------------- Using system modules: Using mpi.h: # 1 "/usr/local/petsc/r/include/mpi.h" 1 ----------------------------------------- Using libraries: -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib -L/usr/local/share/petsc/src/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 -L/opt/rh/devtoolset-7/root/usr/lib64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -ldl ------------------------------------------ Using mpiexec: /usr/local/petsc/r/bin/mpiexec ------------------------------------------ Using MAKE: /usr/bin/gmake Default MAKEFLAGS: MAKE_NP:4 MAKE_LOAD:6.0 MAKEFLAGS: --no-print-directory -- PETSC_ARCH=arch-linux-c-opt PETSC_DIR=/usr/local/share/petsc/src ========================================== /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V= libs /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch=arch-linux-c-opt CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o CXX arch-linux-c-opt/obj/sys/objects/device/interface/global_dcontext.o CXX arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function util::compressed_pair pair_{}; ^ detected during instantiation of class "Petsc::ObjectPool [with T=_p_PetscDeviceContext, Constructor=PetscDeviceContextConstructor]" at line 61 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx" CXX arch-linux-c-opt/obj/sys/objects/device/interface/memory.o compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx (code 2) gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o] Error 2 gmake[3]: *** Waiting for unfinished jobs.... /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function util::compressed_pair pair_{}; ^ detected during instantiation of class "Petsc::ObjectPool [with T=_n_PetscEvent, Constructor=PetscEventConstructor]" at line 54 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx" compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx (code 2) gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o] Error 2 gmake[2]: *** [libs] Error 2 **************************ERROR************************************* Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** make[1]: *** [all] Error 1 make: *** [all] Error 2 *** Using PETSC_DIR=/usr/local/share/petsc/src PETSC_ARCH=arch-linux-c-opt *** *** Installing PETSc at prefix location: /usr/local/petsc/r *** ==================================== Install complete. LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. Confidential - Limited Access and Use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1167 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 6711872 bytes Desc: configure.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log.petscbuild Type: application/octet-stream Size: 89084 bytes Desc: log.petscbuild URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petscbuild.sh Type: application/octet-stream Size: 1148 bytes Desc: petscbuild.sh URL: From bsmith at petsc.dev Wed Jul 19 16:02:28 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 19 Jul 2023 17:02:28 -0400 Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC In-Reply-To: References: Message-ID: Do you need C++, you can configure --with-cxx=0 if you do not need it. You can also try the main branch of PETSc or slightly different versions of the compiler. Barry > On Jul 19, 2023, at 4:31 PM, Robert Crockett via petsc-users wrote: > > Hello, > I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. > See the attached for more information. > The file petscbuild.sh is the script used to configure and build > The file configure.log is output by PETSc > The file log.petscbuild is the output of the config & build script. > > Can you please help find a work-around? > Best, > Robert > > PS. Some related links I found in searching on this issue. > The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. > https://community.intel.com/t5/Intel-C-Compiler/Default-constructor-of-variant-is-deleted/m-p/1156212?profile.language=en > https://mediatum.ub.tum.de/doc/1555265/1555265.pdf > > Robert Crockett > Plasma Simulation Engineer | OCTO - Computational Products > P: 617.648.8349 M: 415.205.4567 > > LAM RESEARCH > 4650 Cushing Pkwy, Fremont CA 94538 USA > lamresearch.com > > > ----------------------------------------- > Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic > mpicc -show: icc -fPIC -wd1572 -O3 -mtune=generic -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > C compiler version: icc (ICC) 18.0.5 20180823 > Using C++ compile: /usr/local/petsc/r/bin/mpicxx -o .o -c -wd1572 -O3 -mtune=generic -std=c++14 -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include > mpicxx -show: icpc -wd1572 -O3 -mtune=generic -std=c++14 -fPIC -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > C++ compiler version: icpc (ICC) 18.0.5 20180823 > Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include > mpif90 -show: gfortran -fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune=generic -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > Fortran compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) > ----------------------------------------- > Using C/C++ linker: /usr/local/petsc/r/bin/mpicc > Using C/C++ flags: -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic > Using Fortran linker: /usr/local/petsc/r/bin/mpif90 > Using Fortran flags: -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic > ----------------------------------------- > Using system modules: > Using mpi.h: # 1 "/usr/local/petsc/r/include/mpi.h" 1 > ----------------------------------------- > Using libraries: -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib -L/usr/local/share/petsc/src/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 -L/opt/rh/devtoolset-7/root/usr/lib64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -ldl > ------------------------------------------ > Using mpiexec: /usr/local/petsc/r/bin/mpiexec > ------------------------------------------ > Using MAKE: /usr/bin/gmake > Default MAKEFLAGS: MAKE_NP:4 MAKE_LOAD:6.0 MAKEFLAGS: --no-print-directory -- PETSC_ARCH=arch-linux-c-opt PETSC_DIR=/usr/local/share/petsc/src > ========================================== > /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V= libs > /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch=arch-linux-c-opt > CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o > CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o > CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o > CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o > CXX arch-linux-c-opt/obj/sys/objects/device/interface/global_dcontext.o > CXX arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o > /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function > util::compressed_pair pair_{}; > ^ > detected during instantiation of class "Petsc::ObjectPool [with T=_p_PetscDeviceContext, Constructor=PetscDeviceContextConstructor]" at line 61 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx" > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/memory.o > compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx (code 2) > gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o] Error 2 > gmake[3]: *** Waiting for unfinished jobs.... > /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function > util::compressed_pair pair_{}; > ^ > detected during instantiation of class "Petsc::ObjectPool [with T=_n_PetscEvent, Constructor=PetscEventConstructor]" at line 54 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx" > > compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx (code 2) > gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o] Error 2 > gmake[2]: *** [libs] Error 2 > **************************ERROR************************************* > Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log > Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > ******************************************************************** > make[1]: *** [all] Error 1 > make: *** [all] Error 2 > *** Using PETSC_DIR=/usr/local/share/petsc/src PETSC_ARCH=arch-linux-c-opt *** > *** Installing PETSc at prefix location: /usr/local/petsc/r *** > ==================================== > Install complete. > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > Confidential ? Limited Access and Use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jul 19 16:06:55 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 Jul 2023 16:06:55 -0500 (CDT) Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC In-Reply-To: References: Message-ID: you can try --with-cxx-dialect=11 and see if that works. with -with-cxx=0 - --download-suitesparse [and other pkgs that require cxx] won't work - and would need to be installed separately Jacob, One more use case for --with-cxx-bindings=0 Satish On Wed, 19 Jul 2023, Barry Smith wrote: > > Do you need C++, you can configure --with-cxx=0 if you do not need it. > > You can also try the main branch of PETSc or slightly different versions of the compiler. > > Barry > > > > > On Jul 19, 2023, at 4:31 PM, Robert Crockett via petsc-users wrote: > > > > Hello, > > I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. > > See the attached for more information. > > The file petscbuild.sh is the script used to configure and build > > The file configure.log is output by PETSc > > The file log.petscbuild is the output of the config & build script. > > > > Can you please help find a work-around? > > Best, > > Robert > > > > PS. Some related links I found in searching on this issue. > > The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. > > https://community.intel.com/t5/Intel-C-Compiler/Default-constructor-of-variant-is-deleted/m-p/1156212?profile.language=en > > https://mediatum.ub.tum.de/doc/1555265/1555265.pdf > > > > Robert Crockett > > Plasma Simulation Engineer | OCTO - Computational Products > > P: 617.648.8349 M: 415.205.4567 > > > > LAM RESEARCH > > 4650 Cushing Pkwy, Fremont CA 94538 USA > > lamresearch.com > > > > > > ----------------------------------------- > > Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic > > mpicc -show: icc -fPIC -wd1572 -O3 -mtune=generic -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > > C compiler version: icc (ICC) 18.0.5 20180823 > > Using C++ compile: /usr/local/petsc/r/bin/mpicxx -o .o -c -wd1572 -O3 -mtune=generic -std=c++14 -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include > > mpicxx -show: icpc -wd1572 -O3 -mtune=generic -std=c++14 -fPIC -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > > C++ compiler version: icpc (ICC) 18.0.5 20180823 > > Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include > > mpif90 -show: gfortran -fPIC -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune=generic -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi > > Fortran compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) > > ----------------------------------------- > > Using C/C++ linker: /usr/local/petsc/r/bin/mpicc > > Using C/C++ flags: -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic > > Using Fortran linker: /usr/local/petsc/r/bin/mpif90 > > Using Fortran flags: -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic > > ----------------------------------------- > > Using system modules: > > Using mpi.h: # 1 "/usr/local/petsc/r/include/mpi.h" 1 > > ----------------------------------------- > > Using libraries: -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib -L/usr/local/share/petsc/src/arch-linux-c-opt/lib -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 -L/opt/rh/devtoolset-7/root/usr/lib64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/ipp/lib/intel64 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_l ibraries _2018.5.274/linux/mkl/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64/gcc4.7 -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/daal/lib/intel64_lin -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch - lscotche rr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -ldl > > ------------------------------------------ > > Using mpiexec: /usr/local/petsc/r/bin/mpiexec > > ------------------------------------------ > > Using MAKE: /usr/bin/gmake > > Default MAKEFLAGS: MAKE_NP:4 MAKE_LOAD:6.0 MAKEFLAGS: --no-print-directory -- PETSC_ARCH=arch-linux-c-opt PETSC_DIR=/usr/local/share/petsc/src > > ========================================== > > /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V= libs > > /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch=arch-linux-c-opt > > CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o > > CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/global_dcontext.o > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o > > /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function > > util::compressed_pair pair_{}; > > ^ > > detected during instantiation of class "Petsc::ObjectPool [with T=_p_PetscDeviceContext, Constructor=PetscDeviceContextConstructor]" at line 61 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx" > > > > CXX arch-linux-c-opt/obj/sys/objects/device/interface/memory.o > > compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx (code 2) > > gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o] Error 2 > > gmake[3]: *** Waiting for unfinished jobs.... > > /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function > > util::compressed_pair pair_{}; > > ^ > > detected during instantiation of class "Petsc::ObjectPool [with T=_n_PetscEvent, Constructor=PetscEventConstructor]" at line 54 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx" > > > > compilation aborted for /usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx (code 2) > > gmake[3]: *** [arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o] Error 2 > > gmake[2]: *** [libs] Error 2 > > **************************ERROR************************************* > > Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log > > Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > > ******************************************************************** > > make[1]: *** [all] Error 1 > > make: *** [all] Error 2 > > *** Using PETSC_DIR=/usr/local/share/petsc/src PETSC_ARCH=arch-linux-c-opt *** > > *** Installing PETSc at prefix location: /usr/local/petsc/r *** > > ==================================== > > Install complete. > > > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > > > Confidential ? Limited Access and Use > > > > > > From bubu.cattaneo at gmail.com Wed Jul 19 21:04:25 2023 From: bubu.cattaneo at gmail.com (Alberto Cattaneo) Date: Wed, 19 Jul 2023 21:04:25 -0500 Subject: [petsc-users] New to using PETSc, running into issue Message-ID: Greetings I hope this email reaches you well. I'm new to using PETSc and so I thought a good way to familiarize myself with the tools would be to noodle around with a simple problem and see how different setups of the same problem play out. I thought it would be prudent to try ex50.c with different linear solvers to see how they compare on a simple problem that was easy enough to tweak. However I'm having some trouble, when I try to set the KSP type the only one that seems to work is KSP_CG_SYMMETRIC, all others throw up an error like this /home/cattaneo/petsc-3.19.1/include/petscksp.h:59:23: error: incompatible type for argument 2 of ?KSPCGSetType? 59 | #define KSPTCQMR "tcqmr" | ^~~~~~~ | | | const char * /home/cattaneo/petsc-3.19.1/include/petscerror.h:473:28: note: in definition of macro ?PetscCall? 473 | ierr_petsc_call_q_ = __VA_ARGS__; \ | ^~~~~~~~~~~ ex50B.c:56:29: note: in expansion of macro ?KSPTCQMR? 56 | PetscCall(KSPCGSetType(ksp,KSPTCQMR)); | ^~~~~~~~ In file included from ex50B.c:25: /home/cattaneo/petsc-3.19.1/include/petscksp.h:884:47: note: expected ?KSPCGType? but argument is of type ?const char *? 884 | PETSC_EXTERN PetscErrorCode KSPCGSetType(KSP, KSPCGType); When trying to make the file. I suspect I'm missing something fundamental, but I can't seem to chase whatever it is down. I am using PETSc on WSL2 on Windows 10 which I know is not the standard, but so far it hasn't seemed to cause trouble. I'm confused because I'm using the same make that is used for all the examples, and it doesn't seem to have an issue using KSPGMRES for ex62 as an example. Sorry if this is a basic question, I'm just a little unsure where to turn as I don't really know anyone that uses PETSc. Thank you for your time. Respectfully: Alberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Jul 20 00:26:55 2023 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 20 Jul 2023 07:26:55 +0200 Subject: [petsc-users] New to using PETSc, running into issue In-Reply-To: References: Message-ID: You are mixing up KSPSetType() and KSPCGSetType(). KSPTCQMR can be passed as an argument of KSPSetType(), not KSPCGSetType(). The former has an argument of type KSPType, which is a string, while the latter has an argument of type KSPCGType, which is an enumerate. Jose > El 20 jul 2023, a las 4:04, Alberto Cattaneo escribi?: > > Greetings > I hope this email reaches you well. I'm new to using PETSc and so I thought a good way to familiarize myself with the tools would be to noodle around with a simple problem and see how different setups of the same problem play out. I thought it would be prudent to try ex50.c with different linear solvers to see how they compare on a simple problem that was easy enough to tweak. However I'm having some trouble, when I try to set the KSP type the only one that seems to work is KSP_CG_SYMMETRIC, all others throw up an error like this > > /home/cattaneo/petsc-3.19.1/include/petscksp.h:59:23: error: incompatible type for argument 2 of ?KSPCGSetType? > 59 | #define KSPTCQMR "tcqmr" > | ^~~~~~~ > | | > | const char * > /home/cattaneo/petsc-3.19.1/include/petscerror.h:473:28: note: in definition of macro ?PetscCall? > 473 | ierr_petsc_call_q_ = __VA_ARGS__; \ > | ^~~~~~~~~~~ > ex50B.c:56:29: note: in expansion of macro ?KSPTCQMR? > 56 | PetscCall(KSPCGSetType(ksp,KSPTCQMR)); > | ^~~~~~~~ > In file included from ex50B.c:25: > /home/cattaneo/petsc-3.19.1/include/petscksp.h:884:47: note: expected ?KSPCGType? but argument is of type ?const char *? > 884 | PETSC_EXTERN PetscErrorCode KSPCGSetType(KSP, KSPCGType); > > When trying to make the file. I suspect I'm missing something fundamental, but I can't seem to chase whatever it is down. I am using PETSc on WSL2 on Windows 10 which I know is not the standard, but so far it hasn't seemed to cause trouble. I'm confused because I'm using the same make that is used for all the examples, and it doesn't seem to have an issue using KSPGMRES for ex62 as an example. Sorry if this is a basic question, I'm just a little unsure where to turn as I don't really know anyone that uses PETSc. Thank you for your time. > Respectfully: > Alberto > From daniel.stone at opengosim.com Thu Jul 20 04:10:26 2023 From: daniel.stone at opengosim.com (Daniel Stone) Date: Thu, 20 Jul 2023 10:10:26 +0100 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> Message-ID: Hi All, Many thanks for the detailed explainations and ideas! I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: -------------------------------- FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' hypre__global_error = 0; ^ ~ C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' __VA_ARGS__; \ ^~~~~~~~~~~ C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' hypre__global_error = 0; ^ ~ C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' __VA_ARGS__; \ ^~~~~~~~~~~ 2 errors generated. make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 make[3]: *** Waiting for unfinished jobs.... FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' **************************ERROR************************************* Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 ---------------------------------------- But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, which I didn't change: # Configuration options option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) [....] checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. Thanks again, Daniel On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users < petsc-users at mcs.anl.gov> wrote: > I think it should work with static libraries and 64bit compilers. > > That's how I think --download-f2cblaslapack [etc] work. > > Also it works with MS-MPI - even-though its a dll install, the library > stub provides this symbol somehow.. > > balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 > $ nm -Ao msmpi.lib |grep " MPI_Init" > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > > However - if the library symbol is somehow mangled - this configure mode > of checking library functions will fail. > > Checking PETSc dll build: > > balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > libpetsc.lib:libpetsc.dll:0000000000000000 I > __imp_MatCreateSeqAIJWithArrays > libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays > > It also has the unmangled symbol - so I guess this mode can work generally > with dlls. > > Satish > > > On Wed, 19 Jul 2023, Barry Smith wrote: > > > > > Satish, > > > > So it will always fail on Windows with Windows compilers (both with > static and shared libraries)? Is this true for all PETSc external packages? > If so, why does the installation documentation say that some external > packages can work with Windows compilers? (Presumably PETSc cannot since > the configure tests will fail). > > > > Barry > > > > > > > On Jul 19, 2023, at 11:40 AM, Satish Balay wrote: > > > > > > BTW: Some explanation of configure: > > > > > > It attempts the following on linux: > > > > > >>>>>>> > > > Source: > > > #include "confdefs.h" > > > #include "conffix.h" > > > /* Override any gcc2 internal prototype to avoid an error. */ > > > char HYPRE_IJMatrixCreate(); > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > int main(void) { > > > _check_HYPRE_IJMatrixCreate(); > > > return 0; > > > } > > > <<<<<<< > > > > > > Note - it does not include 'HYPRE.h' here - but redefines the > prototype as 'char HYPRE_IJMatrixCreate(); > > > > > > Compiling it manually: > > > > > >>>>> > > > [balay at pj01 petsc]$ cat conftest.c > > > char HYPRE_IJMatrixCreate(); > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > int main(void) { > > > _check_HYPRE_IJMatrixCreate(); > > > return 0; > > > } > > > [balay at pj01 petsc]$ gcc -c conftest.c > > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > > > conftest.o: U HYPRE_IJMatrixCreate > > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep > HYPRE_IJMatrixCreate > > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T > HYPRE_IJMatrixCreate > > > [balay at pj01 petsc]$ > > > <<<< > > > > > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T > HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure > succeeds! > > > > > >>>>>>> > > > [balay at pj01 petsc]$ gcc -o conftest conftest.o > arch-linux-c-debug/lib/libHYPRE.so > > > [balay at pj01 petsc]$ echo $? > > > 0 > > > <<<<< > > > > > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) > etc..] - this might not match - resulting in link failures. > > > > > > Satish > > > > > > > > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > > > > > >> You could try skipping this test [and assume --with-hypre-include and > --with-hypre-lib options are correct] - and see if this works. > > >> > > >> diff --git a/config/BuildSystem/config/packages/hypre.py > b/config/BuildSystem/config/packages/hypre.py > > >> index 5bc88322aa2..2d6c7932e17 100644 > > >> --- a/config/BuildSystem/config/packages/hypre.py > > >> +++ b/config/BuildSystem/config/packages/hypre.py > > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > > >> self.requiresversion = 1 > > >> self.gitcommit = 'v'+self.version > > >> self.download = ['git:// > https://github.com/hypre-space/hypre',' > https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > > >> - self.functions = ['HYPRE_IJMatrixCreate'] > > >> + self.functions = [] > > >> self.includes = ['HYPRE.h'] > > >> self.liblist = [['libHYPRE.a']] > > >> self.buildLanguages = ['C','Cxx'] > > >> > > >> > > >> Satish > > >> > > >> > > >> On Wed, 19 Jul 2023, Barry Smith wrote: > > >> > > >>> > > >>> You don't indicate what type of libraries you built hypre with; > static or shared. My guess is you ended up with shared > > >>> > > >>> I think the answer to your difficulty is hidden in __cdecl (Satish > will know much better than me). When you are looking for symbols in Windows > shared libraries you have to prepend something to the function prototype to > have it successfully found. For example the PETSc include files have these > things __declspec(dllimport) The configure test fails because it does not > provide the needed prototype. Likely you built PTScotch with static > libraries so no problem. > > >>> > > >>> The simplest fix would be to build static hypre libraries. I think > it is a major project to get PETSc configure and macro system to work > properly with external packages that are in Windows shared libraries since > more use of __declspec would be needed. > > >>> > > >>> Barry > > >>> > > >>> The PETSc installation instructions should probably say something > about external packages with Windows shared libraries. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone < > daniel.stone at opengosim.com> wrote: > > >>>> > > >>>> Hello, > > >>>> > > >>>> I'm working on getting a petsc build running on windows. One > necessary package to include is Hypre. I've been able to build Hypre > seperately using cmake, and confirmed that the library works > > >>>> by setting up a VS project to run some of the example programs. > > >>>> > > >>>> My attempted petsc build is being done through cygwin. I've been > able to (with varying degrees of difficulty), build a fairly plain petsc, > and one that downloads and builds ptscotch (after some modifications > > >>>> to both ptscotch and the config script). I am now attempting to > include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note > that the same compilers are being used for both Hypre and for petsc > > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after > again varying amounts of pain to work around their awkwardness with the > config script). > > >>>> > > >>>> I'm seeing a problem when the config script does some tests on the > included hypre lib. The source code looks like: > > >>>> > > >>>> #include "confdefs.h" > > >>>> #include "conffix.h" > > >>>> /* Override any gcc2 internal prototype to avoid an error. */ > > >>>> > > >>>> #include "HYPRE.h" > > >>>> > > >>>> char HYPRE_IJMatrixCreate(); > > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); > } > > >>>> > > >>>> int main() { > > >>>> _check_HYPRE_IJMatrixCreate();; > > >>>> return 0; > > >>>> } > > >>>> > > >>>> > > >>>> As I understand this is a fairly standard type of stub program used > by the config script to check that it is able to link to certain symbols in > given libraries. Tests like this have succeeded in my builds that > > >>>> include PTScotch. > > >>>> > > >>>> I keep getting a linker error with the above test, including if I > seperate it out and try to build it seperately: > > >>>> > > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" > .... > > >>>> > > >>>> Ok, it looks like a problem with either the library or linker > commands. But here's the interesting thing - If I transplant this code into > VS, with the same project setting that allows it to build the much more > > >>>> nontrivial Hypre example programs, I get the same error: > > >>>> > > >>>> Error LNK2001 unresolved external symbol "char __cdecl > HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 > C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > >>>> > > >>>> So it seems like there might be something about this type of stub > program that is not working with my Hypre library. I don't fully understand > this program - it's able to call the function with no arguments, but > > >>>> it also needs to be linked against a library containing the > function, apparently by wrapping it in a static void function? Not > something I've seen before. > > >>>> > > >>>> Does anyone have any insight into what might be going wrong - or > really just any explaination of how the stub program works so I can figure > out why it isn't in this case? > > >>>> > > >>>> Many thanks, > > >>>> > > >>>> Daniel > > >>> > > >>> > > >> > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.stone at opengosim.com Thu Jul 20 05:25:13 2023 From: daniel.stone at opengosim.com (Daniel Stone) Date: Thu, 20 Jul 2023 11:25:13 +0100 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> Message-ID: Ok, so the test does exhibit different errors when run through the config script, compared to my attempts to isolate it: ---------------------------- Executing: [compilation command] successful compile: Source: #include "confdefs.h" #include "conffix.h" /* Override any gcc2 internal prototype to avoid an error. */ char HYPRE_IJMatrixCreate(); static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } int main() { _check_HYPRE_IJMatrixCreate();; return 0; } Executing: [linker command] stdout: LINK : warning LNK4098: defaultlib 'msvcrtd.lib' conflicts with use of other libs; use /NODEFAULTLIB:library LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrmm.c.obj)' LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrmv.c.obj)' LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(lsame.c.obj)' LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(xerbla.c.obj)' LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(lsame.c.obj)' LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrti2.c.obj)' [etc...] HYPRE.lib(par_vector.c.obj) : error LNK2019: unresolved external symbol __imp__wassert referenced in function hypre_ParVectorDestroy HYPRE.lib(vector.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(par_csr_matop.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(par_csr_communication.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(csr_matop.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(merge_sort.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(memory.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(IJMatrix_parcsr.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(par_csr_matrix.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(csr_matrix.c.obj) : error LNK2001: unresolved external symbol __imp__wassert HYPRE.lib(memory.c.obj) : error LNK2019: unresolved external symbol __imp_realloc referenced in function hypre_ReAlloc HYPRE.lib(par_vector.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(vector.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(IJMatrix.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(par_csr_matrix.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(csr_matrix.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(new_commpkg.c.obj) : error LNK2001: unresolved external symbol __imp_fopen HYPRE.lib(printf.c.obj) : error LNK2019: unresolved external symbol __imp___stdio_common_vfscanf referenced in function _vfscanf_l HYPRE.lib(printf.c.obj) : error LNK2019: unresolved external symbol __imp___stdio_common_vsscanf referenced in function _vsscanf_l HYPRE.lib(mmio.c.obj) : error LNK2001: unresolved external symbol __imp___stdio_common_vsscanf HYPRE.lib(mmio.c.obj) : error LNK2019: unresolved external symbol __imp_fgets referenced in function hypre_mm_read_banner HYPRE.lib(mmio.c.obj) : error LNK2019: unresolved external symbol __imp_tolower referenced in function hypre_mm_read_banner C:\cygwin64\tmp\PE0681~1\CONFIG~1.LIB\conftest.exe : fatal error LNK1120: 7 unresolved externals icx: error: linker command failed with exit code 1120 (use -v to see invocation) Possible ERROR while running linker: exit code 96 [more warnings etc] ------------------------- I've looked at the oneapi compiler documentation, and it seems like "nodefaultlib", "nostdlib", etc, options only work for linux versions of the compiler, so I'm not sure what to do about the first warning. From the errors, it looks like some core c or c++ library that hypre depends on isn't visible. I had some similar issues with ptscotch - but in that case I didn't have the warnings, and the errors gave me the names of libraries that were missing, which I lilnked in using the --cflags option (maybe --cc-linker-flags would have been neater, but it worked. I've tried both in order to try to get the above working). I can go into detail about the compile and linker commands if needed; I'd have to explain more about my choices for --cflags, etc too. I wonder if any of the above output shines any light on the hypre-is-shared-library hypothesis. Thanks, Daniel On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users < petsc-users at mcs.anl.gov> wrote: > I think it should work with static libraries and 64bit compilers. > > That's how I think --download-f2cblaslapack [etc] work. > > Also it works with MS-MPI - even-though its a dll install, the library > stub provides this symbol somehow.. > > balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 > $ nm -Ao msmpi.lib |grep " MPI_Init" > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > > However - if the library symbol is somehow mangled - this configure mode > of checking library functions will fail. > > Checking PETSc dll build: > > balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > libpetsc.lib:libpetsc.dll:0000000000000000 I > __imp_MatCreateSeqAIJWithArrays > libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays > > It also has the unmangled symbol - so I guess this mode can work generally > with dlls. > > Satish > > > On Wed, 19 Jul 2023, Barry Smith wrote: > > > > > Satish, > > > > So it will always fail on Windows with Windows compilers (both with > static and shared libraries)? Is this true for all PETSc external packages? > If so, why does the installation documentation say that some external > packages can work with Windows compilers? (Presumably PETSc cannot since > the configure tests will fail). > > > > Barry > > > > > > > On Jul 19, 2023, at 11:40 AM, Satish Balay wrote: > > > > > > BTW: Some explanation of configure: > > > > > > It attempts the following on linux: > > > > > >>>>>>> > > > Source: > > > #include "confdefs.h" > > > #include "conffix.h" > > > /* Override any gcc2 internal prototype to avoid an error. */ > > > char HYPRE_IJMatrixCreate(); > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > int main(void) { > > > _check_HYPRE_IJMatrixCreate(); > > > return 0; > > > } > > > <<<<<<< > > > > > > Note - it does not include 'HYPRE.h' here - but redefines the > prototype as 'char HYPRE_IJMatrixCreate(); > > > > > > Compiling it manually: > > > > > >>>>> > > > [balay at pj01 petsc]$ cat conftest.c > > > char HYPRE_IJMatrixCreate(); > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > int main(void) { > > > _check_HYPRE_IJMatrixCreate(); > > > return 0; > > > } > > > [balay at pj01 petsc]$ gcc -c conftest.c > > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > > > conftest.o: U HYPRE_IJMatrixCreate > > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep > HYPRE_IJMatrixCreate > > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T > HYPRE_IJMatrixCreate > > > [balay at pj01 petsc]$ > > > <<<< > > > > > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T > HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure > succeeds! > > > > > >>>>>>> > > > [balay at pj01 petsc]$ gcc -o conftest conftest.o > arch-linux-c-debug/lib/libHYPRE.so > > > [balay at pj01 petsc]$ echo $? > > > 0 > > > <<<<< > > > > > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) > etc..] - this might not match - resulting in link failures. > > > > > > Satish > > > > > > > > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > > > > > >> You could try skipping this test [and assume --with-hypre-include and > --with-hypre-lib options are correct] - and see if this works. > > >> > > >> diff --git a/config/BuildSystem/config/packages/hypre.py > b/config/BuildSystem/config/packages/hypre.py > > >> index 5bc88322aa2..2d6c7932e17 100644 > > >> --- a/config/BuildSystem/config/packages/hypre.py > > >> +++ b/config/BuildSystem/config/packages/hypre.py > > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > > >> self.requiresversion = 1 > > >> self.gitcommit = 'v'+self.version > > >> self.download = ['git:// > https://github.com/hypre-space/hypre',' > https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > > >> - self.functions = ['HYPRE_IJMatrixCreate'] > > >> + self.functions = [] > > >> self.includes = ['HYPRE.h'] > > >> self.liblist = [['libHYPRE.a']] > > >> self.buildLanguages = ['C','Cxx'] > > >> > > >> > > >> Satish > > >> > > >> > > >> On Wed, 19 Jul 2023, Barry Smith wrote: > > >> > > >>> > > >>> You don't indicate what type of libraries you built hypre with; > static or shared. My guess is you ended up with shared > > >>> > > >>> I think the answer to your difficulty is hidden in __cdecl (Satish > will know much better than me). When you are looking for symbols in Windows > shared libraries you have to prepend something to the function prototype to > have it successfully found. For example the PETSc include files have these > things __declspec(dllimport) The configure test fails because it does not > provide the needed prototype. Likely you built PTScotch with static > libraries so no problem. > > >>> > > >>> The simplest fix would be to build static hypre libraries. I think > it is a major project to get PETSc configure and macro system to work > properly with external packages that are in Windows shared libraries since > more use of __declspec would be needed. > > >>> > > >>> Barry > > >>> > > >>> The PETSc installation instructions should probably say something > about external packages with Windows shared libraries. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone < > daniel.stone at opengosim.com> wrote: > > >>>> > > >>>> Hello, > > >>>> > > >>>> I'm working on getting a petsc build running on windows. One > necessary package to include is Hypre. I've been able to build Hypre > seperately using cmake, and confirmed that the library works > > >>>> by setting up a VS project to run some of the example programs. > > >>>> > > >>>> My attempted petsc build is being done through cygwin. I've been > able to (with varying degrees of difficulty), build a fairly plain petsc, > and one that downloads and builds ptscotch (after some modifications > > >>>> to both ptscotch and the config script). I am now attempting to > include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note > that the same compilers are being used for both Hypre and for petsc > > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after > again varying amounts of pain to work around their awkwardness with the > config script). > > >>>> > > >>>> I'm seeing a problem when the config script does some tests on the > included hypre lib. The source code looks like: > > >>>> > > >>>> #include "confdefs.h" > > >>>> #include "conffix.h" > > >>>> /* Override any gcc2 internal prototype to avoid an error. */ > > >>>> > > >>>> #include "HYPRE.h" > > >>>> > > >>>> char HYPRE_IJMatrixCreate(); > > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); > } > > >>>> > > >>>> int main() { > > >>>> _check_HYPRE_IJMatrixCreate();; > > >>>> return 0; > > >>>> } > > >>>> > > >>>> > > >>>> As I understand this is a fairly standard type of stub program used > by the config script to check that it is able to link to certain symbols in > given libraries. Tests like this have succeeded in my builds that > > >>>> include PTScotch. > > >>>> > > >>>> I keep getting a linker error with the above test, including if I > seperate it out and try to build it seperately: > > >>>> > > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" > .... > > >>>> > > >>>> Ok, it looks like a problem with either the library or linker > commands. But here's the interesting thing - If I transplant this code into > VS, with the same project setting that allows it to build the much more > > >>>> nontrivial Hypre example programs, I get the same error: > > >>>> > > >>>> Error LNK2001 unresolved external symbol "char __cdecl > HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 > C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > >>>> > > >>>> So it seems like there might be something about this type of stub > program that is not working with my Hypre library. I don't fully understand > this program - it's able to call the function with no arguments, but > > >>>> it also needs to be linked against a library containing the > function, apparently by wrapping it in a static void function? Not > something I've seen before. > > >>>> > > >>>> Does anyone have any insight into what might be going wrong - or > really just any explaination of how the stub program works so I can figure > out why it isn't in this case? > > >>>> > > >>>> Many thanks, > > >>>> > > >>>> Daniel > > >>> > > >>> > > >> > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 20 08:49:03 2023 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 20 Jul 2023 09:49:03 -0400 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> Message-ID: <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed. Barry > On Jul 20, 2023, at 5:10 AM, Daniel Stone wrote: > > Hi All, > > Many thanks for the detailed explainations and ideas! > > I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: > > -------------------------------- > > FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' > hypre__global_error = 0; > ^ ~ > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > __VA_ARGS__; \ > ^~~~~~~~~~~ > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' > hypre__global_error = 0; > ^ ~ > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > __VA_ARGS__; \ > ^~~~~~~~~~~ > 2 errors generated. > make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 > make[3]: *** Waiting for unfinished jobs.... > FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o > CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o > CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o > make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 > make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > **************************ERROR************************************* > Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log > Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > ******************************************************************** > Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 > > ---------------------------------------- > > But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" > cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied > into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. > > > On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, > which I didn't change: > > # Configuration options > option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) > option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) > option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) > [....] > > > checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. > > > > Thanks again, > > Daniel > > > > On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users > wrote: >> I think it should work with static libraries and 64bit compilers. >> >> That's how I think --download-f2cblaslapack [etc] work. >> >> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. >> >> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 >> $ nm -Ao msmpi.lib |grep " MPI_Init" >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized >> >> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. >> >> Checking PETSc dll build: >> >> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib >> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays >> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays >> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays >> >> It also has the unmangled symbol - so I guess this mode can work generally with dlls. >> >> Satish >> >> >> On Wed, 19 Jul 2023, Barry Smith wrote: >> >> > >> > Satish, >> > >> > So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). >> > >> > Barry >> > >> > >> > > On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: >> > > >> > > BTW: Some explanation of configure: >> > > >> > > It attempts the following on linux: >> > > >> > >>>>>>> >> > > Source: >> > > #include "confdefs.h" >> > > #include "conffix.h" >> > > /* Override any gcc2 internal prototype to avoid an error. */ >> > > char HYPRE_IJMatrixCreate(); >> > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >> > > >> > > int main(void) { >> > > _check_HYPRE_IJMatrixCreate(); >> > > return 0; >> > > } >> > > <<<<<<< >> > > >> > > Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); >> > > >> > > Compiling it manually: >> > > >> > >>>>> >> > > [balay at pj01 petsc]$ cat conftest.c >> > > char HYPRE_IJMatrixCreate(); >> > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >> > > >> > > int main(void) { >> > > _check_HYPRE_IJMatrixCreate(); >> > > return 0; >> > > } >> > > [balay at pj01 petsc]$ gcc -c conftest.c >> > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate >> > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate >> > > conftest.o: U HYPRE_IJMatrixCreate >> > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate >> > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate >> > > [balay at pj01 petsc]$ >> > > <<<< >> > > >> > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! >> > > >> > >>>>>>> >> > > [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so >> > > [balay at pj01 petsc]$ echo $? >> > > 0 >> > > <<<<< >> > > >> > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. >> > > >> > > Satish >> > > >> > > >> > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: >> > > >> > >> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. >> > >> >> > >> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py >> > >> index 5bc88322aa2..2d6c7932e17 100644 >> > >> --- a/config/BuildSystem/config/packages/hypre.py >> > >> +++ b/config/BuildSystem/config/packages/hypre.py >> > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): >> > >> self.requiresversion = 1 >> > >> self.gitcommit = 'v'+self.version >> > >> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] >> > >> - self.functions = ['HYPRE_IJMatrixCreate'] >> > >> + self.functions = [] >> > >> self.includes = ['HYPRE.h'] >> > >> self.liblist = [['libHYPRE.a']] >> > >> self.buildLanguages = ['C','Cxx'] >> > >> >> > >> >> > >> Satish >> > >> >> > >> >> > >> On Wed, 19 Jul 2023, Barry Smith wrote: >> > >> >> > >>> >> > >>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared >> > >>> >> > >>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. >> > >>> >> > >>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. >> > >>> >> > >>> Barry >> > >>> >> > >>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone > wrote: >> > >>>> >> > >>>> Hello, >> > >>>> >> > >>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works >> > >>>> by setting up a VS project to run some of the example programs. >> > >>>> >> > >>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications >> > >>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc >> > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). >> > >>>> >> > >>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: >> > >>>> >> > >>>> #include "confdefs.h" >> > >>>> #include "conffix.h" >> > >>>> /* Override any gcc2 internal prototype to avoid an error. */ >> > >>>> >> > >>>> #include "HYPRE.h" >> > >>>> >> > >>>> char HYPRE_IJMatrixCreate(); >> > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >> > >>>> >> > >>>> int main() { >> > >>>> _check_HYPRE_IJMatrixCreate();; >> > >>>> return 0; >> > >>>> } >> > >>>> >> > >>>> >> > >>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that >> > >>>> include PTScotch. >> > >>>> >> > >>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: >> > >>>> >> > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... >> > >>>> >> > >>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more >> > >>>> nontrivial Hypre example programs, I get the same error: >> > >>>> >> > >>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 >> > >>>> >> > >>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but >> > >>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. >> > >>>> >> > >>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? >> > >>>> >> > >>>> Many thanks, >> > >>>> >> > >>>> Daniel >> > >>> >> > >>> >> > >> >> > > >> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Crockett at lamresearch.com Thu Jul 20 09:27:34 2023 From: Robert.Crockett at lamresearch.com (Robert.Crockett at lamresearch.com) Date: Thu, 20 Jul 2023 14:27:34 +0000 Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC In-Reply-To: References: Message-ID: Satish, Unfortunately, --with-cxx-dialect+AD0-11 did not fix the problem. Can you or others on the list tell me the effect on solving large sparse real symmetric and unsymmetric problems, if --with-cxx+AD0-0 and suitesparse is not downloaded? Thanks, Robert Confidential +IBM- Limited Access and Use -----Original Message----- From: Satish Balay +ADw-balay+AEA-mcs.anl.gov+AD4- Sent: Wednesday, July 19, 2023 5:07 PM To: Barry Smith +ADw-bsmith+AEA-petsc.dev+AD4- Cc: Crockett, Robert +ADw-Robert.Crockett+AEA-lamresearch.com+AD4AOw- petsc-users+AEA-mcs.anl.gov Subject: Re: +AFs-petsc-users+AF0- Failing PETSc 3.19.2 compile using ICPC you can try --with-cxx-dialect+AD0-11 and see if that works. with -with-cxx+AD0-0 - --download-suitesparse +AFs-and other pkgs that require cxx+AF0- won't work - and would need to be installed separately Jacob, One more use case for --with-cxx-bindings+AD0-0 Satish On Wed, 19 Jul 2023, Barry Smith wrote: +AD4- +AD4- Do you need C+-+-, you can configure --with-cxx+AD0-0 if you do not need it. +AD4- +AD4- You can also try the main branch of PETSc or slightly different versions of the compiler. +AD4- +AD4- Barry +AD4- +AD4- +AD4- +AD4- +AD4- On Jul 19, 2023, at 4:31 PM, Robert Crockett via petsc-users +ADw-petsc-users+AEA-mcs.anl.gov+AD4- wrote: +AD4- +AD4- +AD4- +AD4- Hello, +AD4- +AD4- I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. +AD4- +AD4- See the attached for more information. +AD4- +AD4- The file petscbuild.sh is the script used to configure and build The +AD4- +AD4- file configure.log is output by PETSc The file log.petscbuild is the +AD4- +AD4- output of the config +ACY- build script. +AD4- +AD4- +AD4- +AD4- Can you please help find a work-around? +AD4- +AD4- Best, +AD4- +AD4- Robert +AD4- +AD4- +AD4- +AD4- PS. Some related links I found in searching on this issue. +AD4- +AD4- The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. +AD4- +AD4- https://nam02.safelinks.protection.outlook.com/?url+AD0-https+ACU-3A+ACU-2F+ACU-2Fco +AD4- +AD4- mmunity.intel.com+ACU-2Ft5+ACU-2FIntel-C-Compiler+ACU-2FDefault-constructor-of-v +AD4- +AD4- ariant-is-deleted+ACU-2Fm-p+ACU-2F1156212+ACU-3Fprofile.language+ACU-3Den+ACY-data+AD0-05+ACU-7C +AD4- +AD4- 01+ACU-7CRobert.Crockett+ACU-40lamresearch.com+ACU-7C616797c1562a4487ab5008db889 +AD4- +AD4- c18bc+ACU-7C918079dbc9024e29b22c9764410d0375+ACU-7C0+ACU-7C0+ACU-7C63825397626768035 +AD4- +AD4- 8+ACU-7CUnknown+ACU-7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi +AD4- +AD4- I6Ik1haWwiLCJXVCI6Mn0+ACU-3D+ACU-7C3000+ACU-7C+ACU-7C+ACU-7C+ACY-sdata+AD0-aLTeNz0m6xZ4df8AyndNe +AD4- +AD4- 38vqD16gbTfWuKHGw735+ACU-2BY+ACU-3D+ACY-reserved+AD0-0 +AD4- +AD4- https://nam02.safelinks.protection.outlook.com/?url+AD0-https+ACU-3A+ACU-2F+ACU-2Fme +AD4- +AD4- diatum.ub.tum.de+ACU-2Fdoc+ACU-2F1555265+ACU-2F1555265.pdf+ACY-data+AD0-05+ACU-7C01+ACU-7CRobert +AD4- +AD4- .Crockett+ACU-40lamresearch.com+ACU-7C616797c1562a4487ab5008db889c18bc+ACU-7C918 +AD4- +AD4- 079dbc9024e29b22c9764410d0375+ACU-7C0+ACU-7C0+ACU-7C638253976267680358+ACU-7CUnknown +AD4- +AD4- +ACU-7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL +AD4- +AD4- CJXVCI6Mn0+ACU-3D+ACU-7C3000+ACU-7C+ACU-7C+ACU-7C+ACY-sdata+AD0-KfN8V+ACU-2BPnJs5kr5itDINRXfrFUOpRX3 +AD4- +AD4- Kok9OtAC4k1pg+ACU-3D+ACY-reserved+AD0-0 +AD4- +AD4- +AD4- +AD4- Robert Crockett +AD4- +AD4- Plasma Simulation Engineer +AHw- OCTO - Computational Products +AD4- +AD4- P: 617.648.8349 M: 415.205.4567 +AD4- +AD4- +AD4- +AD4- LAM RESEARCH +AD4- +AD4- 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com +AD4- +AD4- +ADw-https://www.lamresearch.com/+AD4- +ADw-image001.png+AD4- +AD4- +AD4- +AD4- +AD4- ----------------------------------------- +AD4- +AD4- Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 +AD4- +AD4- -Wno-unknown-pragmas -O3 -mtune+AD0-generic mpicc -show: icc -fPIC +AD4- +AD4- -wd1572 -O3 -mtune+AD0-generic -I/usr/local/petsc/r/include +AD4- +AD4- -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib +AD4- +AD4- -Wl,--enable-new-dtags -lmpi C compiler version: icc (ICC) 18.0.5 +AD4- +AD4- 20180823 Using C+-+- compile: /usr/local/petsc/r/bin/mpicxx -o .o -c +AD4- +AD4- -wd1572 -O3 -mtune+AD0-generic -std+AD0-c+-+-14 +AD4- +AD4- -I/usr/local/share/petsc/src/include +AD4- +AD4- -I/usr/local/share/petsc/src/arch-linux-c-opt/include +AD4- +AD4- -I/usr/local/petsc/r/include mpicxx -show: icpc -wd1572 -O3 +AD4- +AD4- -mtune+AD0-generic -std+AD0-c+-+-14 -fPIC -I/usr/local/petsc/r/include +AD4- +AD4- -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath +AD4- +AD4- -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi +AD4- +AD4- C+-+- compiler version: icpc (ICC) 18.0.5 20180823 +AD4- +AD4- Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune+AD0-generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include +AD4- +AD4- mpif90 -show: gfortran -fPIC -ffree-line-length-none +AD4- +AD4- -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune+AD0-generic +AD4- +AD4- -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include +AD4- +AD4- -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath +AD4- +AD4- -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi Fortran +AD4- +AD4- compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) +AD4- +AD4- ----------------------------------------- +AD4- +AD4- Using C/C+-+- linker: /usr/local/petsc/r/bin/mpicc Using C/C+-+- flags: +AD4- +AD4- -wd1572 -Wno-unknown-pragmas -O3 -mtune+AD0-generic Using Fortran +AD4- +AD4- linker: /usr/local/petsc/r/bin/mpif90 Using Fortran flags: -Wall +AD4- +AD4- -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch +AD4- +AD4- -Wno-unused-dummy-argument -O3 -mtune+AD0-generic +AD4- +AD4- ----------------------------------------- +AD4- +AD4- Using system modules: +AD4- +AD4- Using mpi.h: +ACM- 1 +ACI-/usr/local/petsc/r/include/mpi.h+ACI- 1 +AD4- +AD4- ----------------------------------------- +AD4- +AD4- Using libraries: +AD4- +AD4- -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib +AD4- +AD4- -L/usr/local/share/petsc/src/arch-linux-c-opt/lib +AD4- +AD4- -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib +AD4- +AD4- -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86+AF8-64-redhat-linux +AD4- +AD4- /7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86+AF8-64-redhat-linux/7 +AD4- +AD4- -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 +AD4- +AD4- -L/opt/rh/devtoolset-7/root/usr/lib64 +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/ipp/lib/intel64 +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/ipp/lib/intel64 +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/compiler/lib/intel64+AF8-lin +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/compiler/lib/intel64+AF8-lin +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/mkl/lib/intel64+AF8-lin +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/mkl/lib/intel64+AF8-lin +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/tbb/lib/intel64/gcc4.7 +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/tbb/lib/intel64/gcc4.7 +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/daal/lib/intel64+AF8-lin +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/daal/lib/intel64+AF8-lin +AD4- +AD4- -Wl,-rpath,/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-librar +AD4- +AD4- ies+AF8-2018.5.274/linux/tbb/lib/intel64+AF8-lin/gcc4.4 +AD4- +AD4- -L/work/tools/intel/composer+AF8-xe+AF8-2018+AF8-4/compilers+AF8-and+AF8-libraries+AF8-2018. +AD4- +AD4- 5.274/linux/tbb/lib/intel64+AF8-lin/gcc4.4 +AD4- +AD4- -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib +AD4- +AD4- -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu +AD4- +AD4- -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig +AD4- +AD4- -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps +AD4- +AD4- -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch +AD4- +AD4- -lscotcherr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm +AD4- +AD4- -lgfortran -lm -lgcc+AF8-s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml +AD4- +AD4- -lirng -lstdc+-+- -lm -lipgo -ldecimal -lcilkrts -lgcc+AF8-s -lirc -lirc+AF8-s +AD4- +AD4- -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc+-+- -lm -lipgo +AD4- +AD4- -ldecimal -lcilkrts -lgcc+AF8-s -lirc -lirc+AF8-s -ldl +AD4- +AD4- ------------------------------------------ +AD4- +AD4- Using mpiexec: /usr/local/petsc/r/bin/mpiexec +AD4- +AD4- ------------------------------------------ +AD4- +AD4- Using MAKE: /usr/bin/gmake +AD4- +AD4- Default MAKEFLAGS: MAKE+AF8-NP:4 MAKE+AF8-LOAD:6.0 MAKEFLAGS: +AD4- +AD4- --no-print-directory -- PETSC+AF8-ARCH+AD0-arch-linux-c-opt +AD4- +AD4- PETSC+AF8-DIR+AD0-/usr/local/share/petsc/src +AD4- +AD4- +AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9- +AD4- +AD4- /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V+AD0- libs +AD4- +AD4- /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch+AD0-arch-linux-c-opt +AD4- +AD4- CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o +AD4- +AD4- CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o +AD4- +AD4- CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o +AD4- +AD4- CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o +AD4- +AD4- CXX arch-linux-c-opt/obj/sys/objects/device/interface/global+AF8-dcontext.o +AD4- +AD4- CXX +AD4- +AD4- arch-linux-c-opt/obj/sys/objects/device/interface/mark+AF8-dcontext.o +AD4- +AD4- /usr/local/share/petsc/src/include/petsc/private/cpp/object+AF8-pool.hpp(930): error: the default constructor of +ACI-Petsc::util::compressed+AF8-pair+ADw-PetscDeviceContextConstructor, Petsc::memory::PoolAllocator+AD4AIg- cannot be referenced -- it is a deleted function +AD4- +AD4- util::compressed+AF8-pair+ADw-constructor+AF8-type, allocator+AF8-type+AD4- pair+AF8AewB9ADs- +AD4- +AD4- +AF4- +AD4- +AD4- detected during instantiation of class +ACI-Petsc::ObjectPool+ADw-T, Constructor+AD4- +AFs-with T+AD0AXw-p+AF8-PetscDeviceContext, Constructor+AD0-PetscDeviceContextConstructor+AF0AIg- at line 61 of +ACI-/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx+ACI- +AD4- +AD4- +AD4- +AD4- CXX +AD4- +AD4- arch-linux-c-opt/obj/sys/objects/device/interface/memory.o +AD4- +AD4- compilation aborted for +AD4- +AD4- /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext +AD4- +AD4- .cxx (code 2) +AD4- +AD4- gmake+AFs-3+AF0-: +ACoAKgAq- +AD4- +AD4- +AFs-arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o+AF0- Error +AD4- +AD4- 2 +AD4- +AD4- gmake+AFs-3+AF0-: +ACoAKgAq- Waiting for unfinished jobs.... +AD4- +AD4- /usr/local/share/petsc/src/include/petsc/private/cpp/object+AF8-pool.hpp(930): error: the default constructor of +ACI-Petsc::util::compressed+AF8-pair+ADw-PetscEventConstructor, Petsc::memory::PoolAllocator+AD4AIg- cannot be referenced -- it is a deleted function +AD4- +AD4- util::compressed+AF8-pair+ADw-constructor+AF8-type, allocator+AF8-type+AD4- pair+AF8AewB9ADs- +AD4- +AD4- +AF4- +AD4- +AD4- detected during instantiation of class +ACI-Petsc::ObjectPool+ADw-T, Constructor+AD4- +AFs-with T+AD0AXw-n+AF8-PetscEvent, Constructor+AD0-PetscEventConstructor+AF0AIg- at line 54 of +ACI-/usr/local/share/petsc/src/src/sys/objects/device/interface/mark+AF8-dcontext.cxx+ACI- +AD4- +AD4- +AD4- +AD4- compilation aborted for +AD4- +AD4- /usr/local/share/petsc/src/src/sys/objects/device/interface/mark+AF8-dco +AD4- +AD4- ntext.cxx (code 2) +AD4- +AD4- gmake+AFs-3+AF0-: +ACoAKgAq- +AD4- +AD4- +AFs-arch-linux-c-opt/obj/sys/objects/device/interface/mark+AF8-dcontext.o+AF0- +AD4- +AD4- Error 2 +AD4- +AD4- gmake+AFs-2+AF0-: +ACoAKgAq- +AFs-libs+AF0- Error 2 +AD4- +AD4- +ACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKg-ERROR+ACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACo- +AD4- +AD4- Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log +AD4- +AD4- Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to +AD4- +AD4- petsc-maint+AEA-mcs.anl.gov +ADw-mailto:petsc-maint+AEA-mcs.anl.gov+AD4- +AD4- +AD4- +ACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKgAqACoAKg- +AD4- +AD4- make+AFs-1+AF0-: +ACoAKgAq- +AFs-all+AF0- Error 1 +AD4- +AD4- make: +ACoAKgAq- +AFs-all+AF0- Error 2 +AD4- +AD4- +ACoAKgAq- Using PETSC+AF8-DIR+AD0-/usr/local/share/petsc/src +AD4- +AD4- PETSC+AF8-ARCH+AD0-arch-linux-c-opt +ACoAKgAq- +AD4- +AD4- +ACoAKgAq- Installing PETSc at prefix location: /usr/local/petsc/r +ACoAKgAq- +AD4- +AD4- +AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9AD0APQA9- +AD4- +AD4- Install complete. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, +ACI-E-mail Transmission+ACI-) may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. +AD4- +AD4- +AD4- +AD4- Confidential +IBM- Limited Access and Use +AD4- +AD4- +AD4- +AD4- +ADw-configure.log+AD4APA-log.petscbuild+AD4APA-petscbuild.sh+AD4- +AD4- +AD4- From bsmith at petsc.dev Thu Jul 20 09:44:48 2023 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 20 Jul 2023 10:44:48 -0400 Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC In-Reply-To: References: Message-ID: Suitesparse provides good efficient sequential (non-MPI) sparse direct solvers. Probably a bit more efficient than the sequential sparse direct solvers built into PETSc. So it really depends on how you use PETSc to solve your linear systems, if you use iterative methods, such as algebraic multigrid then it doesn't matter. > On Jul 20, 2023, at 10:27 AM, Robert.Crockett at lamresearch.com wrote: > > Satish, > Unfortunately, --with-cxx-dialect=11 did not fix the problem. > > Can you or others on the list tell me the effect on solving large sparse real symmetric and unsymmetric problems, if --with-cxx=0 and suitesparse is not downloaded? > Thanks, > Robert > > > Confidential ? Limited Access and Use > -----Original Message----- > From: Satish Balay > Sent: Wednesday, July 19, 2023 5:07 PM > To: Barry Smith > Cc: Crockett, Robert ; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Failing PETSc 3.19.2 compile using ICPC > > > you can try --with-cxx-dialect=11 and see if that works. > > with -with-cxx=0 - --download-suitesparse [and other pkgs that require cxx] won't work - and would need to be installed separately > > Jacob, > > One more use case for --with-cxx-bindings=0 > > Satish > > On Wed, 19 Jul 2023, Barry Smith wrote: > >> >> Do you need C++, you can configure --with-cxx=0 if you do not need it. >> >> You can also try the main branch of PETSc or slightly different versions of the compiler. >> >> Barry >> >> >> >>> On Jul 19, 2023, at 4:31 PM, Robert Crockett via petsc-users wrote: >>> >>> Hello, >>> I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. >>> See the attached for more information. >>> The file petscbuild.sh is the script used to configure and build The >>> file configure.log is output by PETSc The file log.petscbuild is the >>> output of the config & build script. >>> >>> Can you please help find a work-around? >>> Best, >>> Robert >>> >>> PS. Some related links I found in searching on this issue. >>> The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. >>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fco >>> mmunity.intel.com%2Ft5%2FIntel-C-Compiler%2FDefault-constructor-of-v >>> ariant-is-deleted%2Fm-p%2F1156212%3Fprofile.language%3Den&data=05%7C >>> 01%7CRobert.Crockett%40lamresearch.com%7C616797c1562a4487ab5008db889 >>> c18bc%7C918079dbc9024e29b22c9764410d0375%7C0%7C0%7C63825397626768035 >>> 8%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi >>> I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aLTeNz0m6xZ4df8AyndNe >>> 38vqD16gbTfWuKHGw735%2BY%3D&reserved=0 >>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fme >>> diatum.ub.tum.de%2Fdoc%2F1555265%2F1555265.pdf&data=05%7C01%7CRobert >>> .Crockett%40lamresearch.com%7C616797c1562a4487ab5008db889c18bc%7C918 >>> 079dbc9024e29b22c9764410d0375%7C0%7C0%7C638253976267680358%7CUnknown >>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL >>> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KfN8V%2BPnJs5kr5itDINRXfrFUOpRX3 >>> Kok9OtAC4k1pg%3D&reserved=0 >>> >>> Robert Crockett >>> Plasma Simulation Engineer | OCTO - Computational Products >>> P: 617.648.8349 M: 415.205.4567 >>> >>> LAM RESEARCH >>> 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com >>> >>> >>> ----------------------------------------- >>> Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 >>> -Wno-unknown-pragmas -O3 -mtune=generic mpicc -show: icc -fPIC >>> -wd1572 -O3 -mtune=generic -I/usr/local/petsc/r/include >>> -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib >>> -Wl,--enable-new-dtags -lmpi C compiler version: icc (ICC) 18.0.5 >>> 20180823 Using C++ compile: /usr/local/petsc/r/bin/mpicxx -o .o -c >>> -wd1572 -O3 -mtune=generic -std=c++14 >>> -I/usr/local/share/petsc/src/include >>> -I/usr/local/share/petsc/src/arch-linux-c-opt/include >>> -I/usr/local/petsc/r/include mpicxx -show: icpc -wd1572 -O3 >>> -mtune=generic -std=c++14 -fPIC -I/usr/local/petsc/r/include >>> -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath >>> -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi >>> C++ compiler version: icpc (ICC) 18.0.5 20180823 >>> Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include >>> mpif90 -show: gfortran -fPIC -ffree-line-length-none >>> -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune=generic >>> -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include >>> -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath >>> -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi Fortran >>> compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) >>> ----------------------------------------- >>> Using C/C++ linker: /usr/local/petsc/r/bin/mpicc Using C/C++ flags: >>> -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic Using Fortran >>> linker: /usr/local/petsc/r/bin/mpif90 Using Fortran flags: -Wall >>> -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch >>> -Wno-unused-dummy-argument -O3 -mtune=generic >>> ----------------------------------------- >>> Using system modules: >>> Using mpi.h: # 1 "/usr/local/petsc/r/include/mpi.h" 1 >>> ----------------------------------------- >>> Using libraries: >>> -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib >>> -L/usr/local/share/petsc/src/arch-linux-c-opt/lib >>> -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib >>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux >>> /7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 >>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 >>> -L/opt/rh/devtoolset-7/root/usr/lib64 >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/ipp/lib/intel64 >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/ipp/lib/intel64 >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/compiler/lib/intel64_lin >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/compiler/lib/intel64_lin >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/mkl/lib/intel64_lin >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/mkl/lib/intel64_lin >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/tbb/lib/intel64/gcc4.7 >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/tbb/lib/intel64/gcc4.7 >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/daal/lib/intel64_lin >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/daal/lib/intel64_lin >>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>> ies_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 >>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>> 5.274/linux/tbb/lib/intel64_lin/gcc4.4 >>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib >>> -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu >>> -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig >>> -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps >>> -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch >>> -lscotcherr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm >>> -lgfortran -lm -lgcc_s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml >>> -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s >>> -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo >>> -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -ldl >>> ------------------------------------------ >>> Using mpiexec: /usr/local/petsc/r/bin/mpiexec >>> ------------------------------------------ >>> Using MAKE: /usr/bin/gmake >>> Default MAKEFLAGS: MAKE_NP:4 MAKE_LOAD:6.0 MAKEFLAGS: >>> --no-print-directory -- PETSC_ARCH=arch-linux-c-opt >>> PETSC_DIR=/usr/local/share/petsc/src >>> ========================================== >>> /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V= libs >>> /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch=arch-linux-c-opt >>> CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o >>> CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o >>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o >>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o >>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/global_dcontext.o >>> CXX >>> arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o >>> /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function >>> util::compressed_pair pair_{}; >>> ^ >>> detected during instantiation of class "Petsc::ObjectPool [with T=_p_PetscDeviceContext, Constructor=PetscDeviceContextConstructor]" at line 61 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx" >>> >>> CXX >>> arch-linux-c-opt/obj/sys/objects/device/interface/memory.o >>> compilation aborted for >>> /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext >>> .cxx (code 2) >>> gmake[3]: *** >>> [arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o] Error >>> 2 >>> gmake[3]: *** Waiting for unfinished jobs.... >>> /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function >>> util::compressed_pair pair_{}; >>> ^ >>> detected during instantiation of class "Petsc::ObjectPool [with T=_n_PetscEvent, Constructor=PetscEventConstructor]" at line 54 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx" >>> >>> compilation aborted for >>> /usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dco >>> ntext.cxx (code 2) >>> gmake[3]: *** >>> [arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o] >>> Error 2 >>> gmake[2]: *** [libs] Error 2 >>> **************************ERROR************************************* >>> Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log >>> Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to >>> petsc-maint at mcs.anl.gov >>> ******************************************************************** >>> make[1]: *** [all] Error 1 >>> make: *** [all] Error 2 >>> *** Using PETSC_DIR=/usr/local/share/petsc/src >>> PETSC_ARCH=arch-linux-c-opt *** >>> *** Installing PETSc at prefix location: /usr/local/petsc/r *** >>> ==================================== >>> Install complete. >>> >>> >>> LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. >>> >>> Confidential ? Limited Access and Use >>> >>> >> >> From balay at mcs.anl.gov Thu Jul 20 10:05:16 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 20 Jul 2023 10:05:16 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> Message-ID: Can check config/BuildSystem/config/packages/hypre.py petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc 'main' branch with 2.29.0 Satish On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote: > > You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed. > > Barry > > > > On Jul 20, 2023, at 5:10 AM, Daniel Stone wrote: > > > > Hi All, > > > > Many thanks for the detailed explainations and ideas! > > > > I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: > > > > -------------------------------- > > > > FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o > > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o > > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o > > CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o > > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' > > hypre__global_error = 0; > > ^ ~ > > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > > __VA_ARGS__; \ > > ^~~~~~~~~~~ > > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' > > hypre__global_error = 0; > > ^ ~ > > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > > __VA_ARGS__; \ > > ^~~~~~~~~~~ > > 2 errors generated. > > make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 > > make[3]: *** Waiting for unfinished jobs.... > > FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o > > CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o > > CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o > > make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > > make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 > > make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > > **************************ERROR************************************* > > Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log > > Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > > ******************************************************************** > > Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 > > > > ---------------------------------------- > > > > But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" > > cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied > > into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. > > > > > > On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, > > which I didn't change: > > > > # Configuration options > > option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) > > option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) > > option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) > > [....] > > > > > > checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. > > > > > > > > Thanks again, > > > > Daniel > > > > > > > > On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users > wrote: > >> I think it should work with static libraries and 64bit compilers. > >> > >> That's how I think --download-f2cblaslapack [etc] work. > >> > >> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. > >> > >> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 > >> $ nm -Ao msmpi.lib |grep " MPI_Init" > >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > >> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > >> > >> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. > >> > >> Checking PETSc dll build: > >> > >> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > >> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > >> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays > >> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays > >> > >> It also has the unmangled symbol - so I guess this mode can work generally with dlls. > >> > >> Satish > >> > >> > >> On Wed, 19 Jul 2023, Barry Smith wrote: > >> > >> > > >> > Satish, > >> > > >> > So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). > >> > > >> > Barry > >> > > >> > > >> > > On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: > >> > > > >> > > BTW: Some explanation of configure: > >> > > > >> > > It attempts the following on linux: > >> > > > >> > >>>>>>> > >> > > Source: > >> > > #include "confdefs.h" > >> > > #include "conffix.h" > >> > > /* Override any gcc2 internal prototype to avoid an error. */ > >> > > char HYPRE_IJMatrixCreate(); > >> > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >> > > > >> > > int main(void) { > >> > > _check_HYPRE_IJMatrixCreate(); > >> > > return 0; > >> > > } > >> > > <<<<<<< > >> > > > >> > > Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); > >> > > > >> > > Compiling it manually: > >> > > > >> > >>>>> > >> > > [balay at pj01 petsc]$ cat conftest.c > >> > > char HYPRE_IJMatrixCreate(); > >> > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >> > > > >> > > int main(void) { > >> > > _check_HYPRE_IJMatrixCreate(); > >> > > return 0; > >> > > } > >> > > [balay at pj01 petsc]$ gcc -c conftest.c > >> > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > >> > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > >> > > conftest.o: U HYPRE_IJMatrixCreate > >> > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate > >> > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate > >> > > [balay at pj01 petsc]$ > >> > > <<<< > >> > > > >> > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! > >> > > > >> > >>>>>>> > >> > > [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so > >> > > [balay at pj01 petsc]$ echo $? > >> > > 0 > >> > > <<<<< > >> > > > >> > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. > >> > > > >> > > Satish > >> > > > >> > > > >> > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > >> > > > >> > >> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. > >> > >> > >> > >> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py > >> > >> index 5bc88322aa2..2d6c7932e17 100644 > >> > >> --- a/config/BuildSystem/config/packages/hypre.py > >> > >> +++ b/config/BuildSystem/config/packages/hypre.py > >> > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > >> > >> self.requiresversion = 1 > >> > >> self.gitcommit = 'v'+self.version > >> > >> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > >> > >> - self.functions = ['HYPRE_IJMatrixCreate'] > >> > >> + self.functions = [] > >> > >> self.includes = ['HYPRE.h'] > >> > >> self.liblist = [['libHYPRE.a']] > >> > >> self.buildLanguages = ['C','Cxx'] > >> > >> > >> > >> > >> > >> Satish > >> > >> > >> > >> > >> > >> On Wed, 19 Jul 2023, Barry Smith wrote: > >> > >> > >> > >>> > >> > >>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared > >> > >>> > >> > >>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. > >> > >>> > >> > >>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. > >> > >>> > >> > >>> Barry > >> > >>> > >> > >>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone > wrote: > >> > >>>> > >> > >>>> Hello, > >> > >>>> > >> > >>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > >> > >>>> by setting up a VS project to run some of the example programs. > >> > >>>> > >> > >>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > >> > >>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > >> > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > >> > >>>> > >> > >>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > >> > >>>> > >> > >>>> #include "confdefs.h" > >> > >>>> #include "conffix.h" > >> > >>>> /* Override any gcc2 internal prototype to avoid an error. */ > >> > >>>> > >> > >>>> #include "HYPRE.h" > >> > >>>> > >> > >>>> char HYPRE_IJMatrixCreate(); > >> > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >> > >>>> > >> > >>>> int main() { > >> > >>>> _check_HYPRE_IJMatrixCreate();; > >> > >>>> return 0; > >> > >>>> } > >> > >>>> > >> > >>>> > >> > >>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > >> > >>>> include PTScotch. > >> > >>>> > >> > >>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > >> > >>>> > >> > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > >> > >>>> > >> > >>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > >> > >>>> nontrivial Hypre example programs, I get the same error: > >> > >>>> > >> > >>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > >> > >>>> > >> > >>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > >> > >>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > >> > >>>> > >> > >>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > >> > >>>> > >> > >>>> Many thanks, > >> > >>>> > >> > >>>> Daniel > >> > >>> > >> > >>> > >> > >> > >> > > > >> > > >> > > From jacob.fai at gmail.com Thu Jul 20 10:07:31 2023 From: jacob.fai at gmail.com (Jacob Faibussowitsch) Date: Thu, 20 Jul 2023 11:07:31 -0400 Subject: [petsc-users] Failing PETSc 3.19.2 compile using ICPC In-Reply-To: References: Message-ID: <30D1C512-C2EB-4814-962B-CDA081FB5F21@gmail.com> Robert, Try this branch jacobf/2023-07-20/icpc-compressed-pair-workaround/release and let me know if that worked. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Jul 20, 2023, at 10:44, Barry Smith wrote: > > > Suitesparse provides good efficient sequential (non-MPI) sparse direct solvers. Probably a bit more efficient than the sequential sparse direct solvers built into PETSc. So it really depends on how you use PETSc to solve your linear systems, if you use iterative methods, such as algebraic multigrid then it doesn't matter. > > > >> On Jul 20, 2023, at 10:27 AM, Robert.Crockett at lamresearch.com wrote: >> >> Satish, >> Unfortunately, --with-cxx-dialect=11 did not fix the problem. >> >> Can you or others on the list tell me the effect on solving large sparse real symmetric and unsymmetric problems, if --with-cxx=0 and suitesparse is not downloaded? >> Thanks, >> Robert >> >> >> Confidential ? Limited Access and Use >> -----Original Message----- >> From: Satish Balay >> Sent: Wednesday, July 19, 2023 5:07 PM >> To: Barry Smith >> Cc: Crockett, Robert ; petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] Failing PETSc 3.19.2 compile using ICPC >> >> >> you can try --with-cxx-dialect=11 and see if that works. >> >> with -with-cxx=0 - --download-suitesparse [and other pkgs that require cxx] won't work - and would need to be installed separately >> >> Jacob, >> >> One more use case for --with-cxx-bindings=0 >> >> Satish >> >> On Wed, 19 Jul 2023, Barry Smith wrote: >> >>> >>> Do you need C++, you can configure --with-cxx=0 if you do not need it. >>> >>> You can also try the main branch of PETSc or slightly different versions of the compiler. >>> >>> Barry >>> >>> >>> >>>> On Jul 19, 2023, at 4:31 PM, Robert Crockett via petsc-users wrote: >>>> >>>> Hello, >>>> I am attempting to build PETSc using the 2018.5.274 Intel compiler suite on CentOS7. I get the below error messages. >>>> See the attached for more information. >>>> The file petscbuild.sh is the script used to configure and build The >>>> file configure.log is output by PETSc The file log.petscbuild is the >>>> output of the config & build script. >>>> >>>> Can you please help find a work-around? >>>> Best, >>>> Robert >>>> >>>> PS. Some related links I found in searching on this issue. >>>> The first references a related compiler bug ticket opened with Intel, though I cannot tell if it was closed or if it applied to my compiler. >>>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fco >>>> mmunity.intel.com%2Ft5%2FIntel-C-Compiler%2FDefault-constructor-of-v >>>> ariant-is-deleted%2Fm-p%2F1156212%3Fprofile.language%3Den&data=05%7C >>>> 01%7CRobert.Crockett%40lamresearch.com%7C616797c1562a4487ab5008db889 >>>> c18bc%7C918079dbc9024e29b22c9764410d0375%7C0%7C0%7C63825397626768035 >>>> 8%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi >>>> I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aLTeNz0m6xZ4df8AyndNe >>>> 38vqD16gbTfWuKHGw735%2BY%3D&reserved=0 >>>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fme >>>> diatum.ub.tum.de%2Fdoc%2F1555265%2F1555265.pdf&data=05%7C01%7CRobert >>>> .Crockett%40lamresearch.com%7C616797c1562a4487ab5008db889c18bc%7C918 >>>> 079dbc9024e29b22c9764410d0375%7C0%7C0%7C638253976267680358%7CUnknown >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL >>>> CJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KfN8V%2BPnJs5kr5itDINRXfrFUOpRX3 >>>> Kok9OtAC4k1pg%3D&reserved=0 >>>> >>>> Robert Crockett >>>> Plasma Simulation Engineer | OCTO - Computational Products >>>> P: 617.648.8349 M: 415.205.4567 >>>> >>>> LAM RESEARCH >>>> 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com >>>> >>>> >>>> ----------------------------------------- >>>> Using C compile: /usr/local/petsc/r/bin/mpicc -o .o -c -wd1572 >>>> -Wno-unknown-pragmas -O3 -mtune=generic mpicc -show: icc -fPIC >>>> -wd1572 -O3 -mtune=generic -I/usr/local/petsc/r/include >>>> -L/usr/local/petsc/r/lib -Wl,-rpath -Wl,/usr/local/petsc/r/lib >>>> -Wl,--enable-new-dtags -lmpi C compiler version: icc (ICC) 18.0.5 >>>> 20180823 Using C++ compile: /usr/local/petsc/r/bin/mpicxx -o .o -c >>>> -wd1572 -O3 -mtune=generic -std=c++14 >>>> -I/usr/local/share/petsc/src/include >>>> -I/usr/local/share/petsc/src/arch-linux-c-opt/include >>>> -I/usr/local/petsc/r/include mpicxx -show: icpc -wd1572 -O3 >>>> -mtune=generic -std=c++14 -fPIC -I/usr/local/petsc/r/include >>>> -L/usr/local/petsc/r/lib -lmpicxx -Wl,-rpath >>>> -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi >>>> C++ compiler version: icpc (ICC) 18.0.5 20180823 >>>> Using Fortran compile: /usr/local/petsc/r/bin/mpif90 -o .o -c -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3 -mtune=generic -I/usr/local/share/petsc/src/include -I/usr/local/share/petsc/src/arch-linux-c-opt/include -I/usr/local/petsc/r/include >>>> mpif90 -show: gfortran -fPIC -ffree-line-length-none >>>> -ffree-line-length-0 -Wno-lto-type-mismatch -O3 -mtune=generic >>>> -I/usr/local/petsc/r/include -I/usr/local/petsc/r/include >>>> -L/usr/local/petsc/r/lib -lmpifort -Wl,-rpath >>>> -Wl,/usr/local/petsc/r/lib -Wl,--enable-new-dtags -lmpi Fortran >>>> compiler version: GNU Fortran (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) >>>> ----------------------------------------- >>>> Using C/C++ linker: /usr/local/petsc/r/bin/mpicc Using C/C++ flags: >>>> -wd1572 -Wno-unknown-pragmas -O3 -mtune=generic Using Fortran >>>> linker: /usr/local/petsc/r/bin/mpif90 Using Fortran flags: -Wall >>>> -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch >>>> -Wno-unused-dummy-argument -O3 -mtune=generic >>>> ----------------------------------------- >>>> Using system modules: >>>> Using mpi.h: # 1 "/usr/local/petsc/r/include/mpi.h" 1 >>>> ----------------------------------------- >>>> Using libraries: >>>> -Wl,-rpath,/usr/local/share/petsc/src/arch-linux-c-opt/lib >>>> -L/usr/local/share/petsc/src/arch-linux-c-opt/lib >>>> -Wl,-rpath,/usr/local/petsc/r/lib -L/usr/local/petsc/r/lib >>>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux >>>> /7 -L/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 >>>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib64 >>>> -L/opt/rh/devtoolset-7/root/usr/lib64 >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/ipp/lib/intel64 >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/ipp/lib/intel64 >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/compiler/lib/intel64_lin >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/compiler/lib/intel64_lin >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/mkl/lib/intel64_lin >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/mkl/lib/intel64_lin >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/tbb/lib/intel64/gcc4.7 >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/tbb/lib/intel64/gcc4.7 >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/daal/lib/intel64_lin >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/daal/lib/intel64_lin >>>> -Wl,-rpath,/work/tools/intel/composer_xe_2018_4/compilers_and_librar >>>> ies_2018.5.274/linux/tbb/lib/intel64_lin/gcc4.4 >>>> -L/work/tools/intel/composer_xe_2018_4/compilers_and_libraries_2018. >>>> 5.274/linux/tbb/lib/intel64_lin/gcc4.4 >>>> -Wl,-rpath,/opt/rh/devtoolset-7/root/usr/lib >>>> -L/opt/rh/devtoolset-7/root/usr/lib -lpetsc -lspqr -lumfpack -lklu >>>> -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig >>>> -lscalapack -lsuperlu -lf2clapack -lf2cblas -lptesmumps >>>> -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch >>>> -lscotcherr -lpthread -lmetis -ldl -lmpifort -lmpi -lgfortran -lm >>>> -lgfortran -lm -lgcc_s -lquadmath -ldl -lmpicxx -lmpi -limf -lsvml >>>> -lirng -lstdc++ -lm -lipgo -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s >>>> -lrt -ldl -lmpicxx -lmpi -limf -lsvml -lirng -lstdc++ -lm -lipgo >>>> -ldecimal -lcilkrts -lgcc_s -lirc -lirc_s -ldl >>>> ------------------------------------------ >>>> Using mpiexec: /usr/local/petsc/r/bin/mpiexec >>>> ------------------------------------------ >>>> Using MAKE: /usr/bin/gmake >>>> Default MAKEFLAGS: MAKE_NP:4 MAKE_LOAD:6.0 MAKEFLAGS: >>>> --no-print-directory -- PETSC_ARCH=arch-linux-c-opt >>>> PETSC_DIR=/usr/local/share/petsc/src >>>> ========================================== >>>> /usr/bin/gmake --print-directory -f gmakefile -j4 -l6.0 V= libs >>>> /opt/mambaforge/bin/python3 ./config/gmakegen.py --petsc-arch=arch-linux-c-opt >>>> CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostcontext.o >>>> CXX arch-linux-c-opt/obj/sys/objects/device/impls/host/hostdevice.o >>>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o >>>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/device.o >>>> CXX arch-linux-c-opt/obj/sys/objects/device/interface/global_dcontext.o >>>> CXX >>>> arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o >>>> /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function >>>> util::compressed_pair pair_{}; >>>> ^ >>>> detected during instantiation of class "Petsc::ObjectPool [with T=_p_PetscDeviceContext, Constructor=PetscDeviceContextConstructor]" at line 61 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext.cxx" >>>> >>>> CXX >>>> arch-linux-c-opt/obj/sys/objects/device/interface/memory.o >>>> compilation aborted for >>>> /usr/local/share/petsc/src/src/sys/objects/device/interface/dcontext >>>> .cxx (code 2) >>>> gmake[3]: *** >>>> [arch-linux-c-opt/obj/sys/objects/device/interface/dcontext.o] Error >>>> 2 >>>> gmake[3]: *** Waiting for unfinished jobs.... >>>> /usr/local/share/petsc/src/include/petsc/private/cpp/object_pool.hpp(930): error: the default constructor of "Petsc::util::compressed_pair" cannot be referenced -- it is a deleted function >>>> util::compressed_pair pair_{}; >>>> ^ >>>> detected during instantiation of class "Petsc::ObjectPool [with T=_n_PetscEvent, Constructor=PetscEventConstructor]" at line 54 of "/usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dcontext.cxx" >>>> >>>> compilation aborted for >>>> /usr/local/share/petsc/src/src/sys/objects/device/interface/mark_dco >>>> ntext.cxx (code 2) >>>> gmake[3]: *** >>>> [arch-linux-c-opt/obj/sys/objects/device/interface/mark_dcontext.o] >>>> Error 2 >>>> gmake[2]: *** [libs] Error 2 >>>> **************************ERROR************************************* >>>> Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log >>>> Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to >>>> petsc-maint at mcs.anl.gov >>>> ******************************************************************** >>>> make[1]: *** [all] Error 1 >>>> make: *** [all] Error 2 >>>> *** Using PETSC_DIR=/usr/local/share/petsc/src >>>> PETSC_ARCH=arch-linux-c-opt *** >>>> *** Installing PETSc at prefix location: /usr/local/petsc/r *** >>>> ==================================== >>>> Install complete. >>>> >>>> >>>> LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. >>>> >>>> Confidential ? Limited Access and Use >>>> >>>> >>> >>> > From balay at mcs.anl.gov Thu Jul 20 10:13:23 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 20 Jul 2023 10:13:23 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> Message-ID: On Thu, 20 Jul 2023, Daniel Stone wrote: > Ok, so the test does exhibit different errors when run through the config > script, compared to my attempts to isolate it: > > ---------------------------- > > Executing: [compilation command] > > successful compile: > Source: > #include "confdefs.h" > #include "conffix.h" > /* Override any gcc2 internal prototype to avoid an error. */ > char HYPRE_IJMatrixCreate(); > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > int main() { > _check_HYPRE_IJMatrixCreate();; > return 0; > } > > Executing: [linker command] > > stdout: > LINK : warning LNK4098: defaultlib 'msvcrtd.lib' conflicts with use of > other libs; use /NODEFAULTLIB:library > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrmm.c.obj)' This error comes up when building different objfiles [or libraries] that go into a single binary with different compiler options - for ex: one with /MD - the other with /MT [each option sets and links with a different default compiler library - resulting in such conflicts] https://learn.microsoft.com/en-us/cpp/build/reference/md-mt-ld-use-run-time-library?view=msvc-170 so the usual fix is to make sure all libraries, objfiles are compiled with the same compiler options Satish > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrmv.c.obj)' > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(lsame.c.obj)' > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(xerbla.c.obj)' > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(lsame.c.obj)' > LINK : warning LNK4286: symbol '__stdio_common_vsprintf' defined in > 'libucrt.lib(output.obj)' is imported by 'HYPRE.lib(dtrti2.c.obj)' > [etc...] > HYPRE.lib(par_vector.c.obj) : error LNK2019: unresolved external symbol > __imp__wassert referenced in function hypre_ParVectorDestroy > HYPRE.lib(vector.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(par_csr_matop.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(par_csr_communication.c.obj) : error LNK2001: unresolved external > symbol __imp__wassert > HYPRE.lib(csr_matop.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(merge_sort.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(memory.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(IJMatrix_parcsr.c.obj) : error LNK2001: unresolved external > symbol __imp__wassert > HYPRE.lib(par_csr_matrix.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(csr_matrix.c.obj) : error LNK2001: unresolved external symbol > __imp__wassert > HYPRE.lib(memory.c.obj) : error LNK2019: unresolved external symbol > __imp_realloc referenced in function hypre_ReAlloc > HYPRE.lib(par_vector.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(vector.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(IJMatrix.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(par_csr_matrix.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(csr_matrix.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(new_commpkg.c.obj) : error LNK2001: unresolved external symbol > __imp_fopen > HYPRE.lib(printf.c.obj) : error LNK2019: unresolved external symbol > __imp___stdio_common_vfscanf referenced in function _vfscanf_l > HYPRE.lib(printf.c.obj) : error LNK2019: unresolved external symbol > __imp___stdio_common_vsscanf referenced in function _vsscanf_l > HYPRE.lib(mmio.c.obj) : error LNK2001: unresolved external symbol > __imp___stdio_common_vsscanf > HYPRE.lib(mmio.c.obj) : error LNK2019: unresolved external symbol > __imp_fgets referenced in function hypre_mm_read_banner > HYPRE.lib(mmio.c.obj) : error LNK2019: unresolved external symbol > __imp_tolower referenced in function hypre_mm_read_banner > C:\cygwin64\tmp\PE0681~1\CONFIG~1.LIB\conftest.exe : fatal error LNK1120: 7 > unresolved externals > icx: error: linker command failed with exit code 1120 (use -v to see > invocation) > Possible ERROR while running linker: exit code 96 > [more warnings etc] > > > ------------------------- > > I've looked at the oneapi compiler documentation, and it seems like > "nodefaultlib", "nostdlib", etc, options only work for linux versions of > the compiler, > so I'm not sure what to do about the first warning. From the errors, it > looks like some core c or c++ library that hypre depends on isn't visible. > I had some similar > issues with ptscotch - but in that case I didn't have the warnings, and the > errors gave me the names of libraries that were missing, which I lilnked in > using the --cflags > option (maybe --cc-linker-flags would have been neater, but it worked. I've > tried both in order to try to get the above working). > > > I can go into detail about the compile and linker commands if needed; I'd > have to explain more about my choices for --cflags, etc too. I wonder if > any of the above output > shines any light on the hypre-is-shared-library hypothesis. > > > Thanks, > > Daniel > > On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > I think it should work with static libraries and 64bit compilers. > > > > That's how I think --download-f2cblaslapack [etc] work. > > > > Also it works with MS-MPI - even-though its a dll install, the library > > stub provides this symbol somehow.. > > > > balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 > > $ nm -Ao msmpi.lib |grep " MPI_Init" > > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > > msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > > > > However - if the library symbol is somehow mangled - this configure mode > > of checking library functions will fail. > > > > Checking PETSc dll build: > > > > balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > > $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > > libpetsc.lib:libpetsc.dll:0000000000000000 I > > __imp_MatCreateSeqAIJWithArrays > > libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays > > > > It also has the unmangled symbol - so I guess this mode can work generally > > with dlls. > > > > Satish > > > > > > On Wed, 19 Jul 2023, Barry Smith wrote: > > > > > > > > Satish, > > > > > > So it will always fail on Windows with Windows compilers (both with > > static and shared libraries)? Is this true for all PETSc external packages? > > If so, why does the installation documentation say that some external > > packages can work with Windows compilers? (Presumably PETSc cannot since > > the configure tests will fail). > > > > > > Barry > > > > > > > > > > On Jul 19, 2023, at 11:40 AM, Satish Balay wrote: > > > > > > > > BTW: Some explanation of configure: > > > > > > > > It attempts the following on linux: > > > > > > > >>>>>>> > > > > Source: > > > > #include "confdefs.h" > > > > #include "conffix.h" > > > > /* Override any gcc2 internal prototype to avoid an error. */ > > > > char HYPRE_IJMatrixCreate(); > > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > > > int main(void) { > > > > _check_HYPRE_IJMatrixCreate(); > > > > return 0; > > > > } > > > > <<<<<<< > > > > > > > > Note - it does not include 'HYPRE.h' here - but redefines the > > prototype as 'char HYPRE_IJMatrixCreate(); > > > > > > > > Compiling it manually: > > > > > > > >>>>> > > > > [balay at pj01 petsc]$ cat conftest.c > > > > char HYPRE_IJMatrixCreate(); > > > > static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > > > > > > > > int main(void) { > > > > _check_HYPRE_IJMatrixCreate(); > > > > return 0; > > > > } > > > > [balay at pj01 petsc]$ gcc -c conftest.c > > > > [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > > > > conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > > > > conftest.o: U HYPRE_IJMatrixCreate > > > > [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep > > HYPRE_IJMatrixCreate > > > > arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T > > HYPRE_IJMatrixCreate > > > > [balay at pj01 petsc]$ > > > > <<<< > > > > > > > > Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T > > HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure > > succeeds! > > > > > > > >>>>>>> > > > > [balay at pj01 petsc]$ gcc -o conftest conftest.o > > arch-linux-c-debug/lib/libHYPRE.so > > > > [balay at pj01 petsc]$ echo $? > > > > 0 > > > > <<<<< > > > > > > > > On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) > > etc..] - this might not match - resulting in link failures. > > > > > > > > Satish > > > > > > > > > > > > On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > > > > > > > >> You could try skipping this test [and assume --with-hypre-include and > > --with-hypre-lib options are correct] - and see if this works. > > > >> > > > >> diff --git a/config/BuildSystem/config/packages/hypre.py > > b/config/BuildSystem/config/packages/hypre.py > > > >> index 5bc88322aa2..2d6c7932e17 100644 > > > >> --- a/config/BuildSystem/config/packages/hypre.py > > > >> +++ b/config/BuildSystem/config/packages/hypre.py > > > >> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > > > >> self.requiresversion = 1 > > > >> self.gitcommit = 'v'+self.version > > > >> self.download = ['git:// > > https://github.com/hypre-space/hypre',' > > https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > > > >> - self.functions = ['HYPRE_IJMatrixCreate'] > > > >> + self.functions = [] > > > >> self.includes = ['HYPRE.h'] > > > >> self.liblist = [['libHYPRE.a']] > > > >> self.buildLanguages = ['C','Cxx'] > > > >> > > > >> > > > >> Satish > > > >> > > > >> > > > >> On Wed, 19 Jul 2023, Barry Smith wrote: > > > >> > > > >>> > > > >>> You don't indicate what type of libraries you built hypre with; > > static or shared. My guess is you ended up with shared > > > >>> > > > >>> I think the answer to your difficulty is hidden in __cdecl (Satish > > will know much better than me). When you are looking for symbols in Windows > > shared libraries you have to prepend something to the function prototype to > > have it successfully found. For example the PETSc include files have these > > things __declspec(dllimport) The configure test fails because it does not > > provide the needed prototype. Likely you built PTScotch with static > > libraries so no problem. > > > >>> > > > >>> The simplest fix would be to build static hypre libraries. I think > > it is a major project to get PETSc configure and macro system to work > > properly with external packages that are in Windows shared libraries since > > more use of __declspec would be needed. > > > >>> > > > >>> Barry > > > >>> > > > >>> The PETSc installation instructions should probably say something > > about external packages with Windows shared libraries. > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone < > > daniel.stone at opengosim.com> wrote: > > > >>>> > > > >>>> Hello, > > > >>>> > > > >>>> I'm working on getting a petsc build running on windows. One > > necessary package to include is Hypre. I've been able to build Hypre > > seperately using cmake, and confirmed that the library works > > > >>>> by setting up a VS project to run some of the example programs. > > > >>>> > > > >>>> My attempted petsc build is being done through cygwin. I've been > > able to (with varying degrees of difficulty), build a fairly plain petsc, > > and one that downloads and builds ptscotch (after some modifications > > > >>>> to both ptscotch and the config script). I am now attempting to > > include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note > > that the same compilers are being used for both Hypre and for petsc > > > >>>> through cygwin - the new intel oneapi compilers (icx and ifx, after > > again varying amounts of pain to work around their awkwardness with the > > config script). > > > >>>> > > > >>>> I'm seeing a problem when the config script does some tests on the > > included hypre lib. The source code looks like: > > > >>>> > > > >>>> #include "confdefs.h" > > > >>>> #include "conffix.h" > > > >>>> /* Override any gcc2 internal prototype to avoid an error. */ > > > >>>> > > > >>>> #include "HYPRE.h" > > > >>>> > > > >>>> char HYPRE_IJMatrixCreate(); > > > >>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); > > } > > > >>>> > > > >>>> int main() { > > > >>>> _check_HYPRE_IJMatrixCreate();; > > > >>>> return 0; > > > >>>> } > > > >>>> > > > >>>> > > > >>>> As I understand this is a fairly standard type of stub program used > > by the config script to check that it is able to link to certain symbols in > > given libraries. Tests like this have succeeded in my builds that > > > >>>> include PTScotch. > > > >>>> > > > >>>> I keep getting a linker error with the above test, including if I > > seperate it out and try to build it seperately: > > > >>>> > > > >>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" > > .... > > > >>>> > > > >>>> Ok, it looks like a problem with either the library or linker > > commands. But here's the interesting thing - If I transplant this code into > > VS, with the same project setting that allows it to build the much more > > > >>>> nontrivial Hypre example programs, I get the same error: > > > >>>> > > > >>>> Error LNK2001 unresolved external symbol "char __cdecl > > HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 > > C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > > > >>>> > > > >>>> So it seems like there might be something about this type of stub > > program that is not working with my Hypre library. I don't fully understand > > this program - it's able to call the function with no arguments, but > > > >>>> it also needs to be linked against a library containing the > > function, apparently by wrapping it in a static void function? Not > > something I've seen before. > > > >>>> > > > >>>> Does anyone have any insight into what might be going wrong - or > > really just any explaination of how the stub program works so I can figure > > out why it isn't in this case? > > > >>>> > > > >>>> Many thanks, > > > >>>> > > > >>>> Daniel > > > >>> > > > >>> > > > >> > > > > > > > > > > > > From bsmith at petsc.dev Thu Jul 20 18:49:44 2023 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 20 Jul 2023 19:49:44 -0400 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> Message-ID: <3D80FBFC-58A1-43E5-AD3D-9D18790796BB@petsc.dev> Satish, I think hypre.py in main needs minversion = 2.29 instead of 2.14 > On Jul 20, 2023, at 11:05 AM, Satish Balay wrote: > > Can check config/BuildSystem/config/packages/hypre.py > > petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc 'main' branch with 2.29.0 > > Satish > > On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote: > >> >> You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed. >> >> Barry >> >> >>> On Jul 20, 2023, at 5:10 AM, Daniel Stone wrote: >>> >>> Hi All, >>> >>> Many thanks for the detailed explainations and ideas! >>> >>> I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: >>> >>> -------------------------------- >>> >>> FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' >>> hypre__global_error = 0; >>> ^ ~ >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' >>> __VA_ARGS__; \ >>> ^~~~~~~~~~~ >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' >>> hypre__global_error = 0; >>> ^ ~ >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' >>> __VA_ARGS__; \ >>> ^~~~~~~~~~~ >>> 2 errors generated. >>> make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 >>> make[3]: *** Waiting for unfinished jobs.... >>> FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o >>> CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o >>> CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o >>> make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' >>> make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 >>> make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' >>> **************************ERROR************************************* >>> Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log >>> Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov >>> ******************************************************************** >>> Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 >>> >>> ---------------------------------------- >>> >>> But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" >>> cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied >>> into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. >>> >>> >>> On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, >>> which I didn't change: >>> >>> # Configuration options >>> option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) >>> option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) >>> option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) >>> [....] >>> >>> >>> checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. >>> >>> >>> >>> Thanks again, >>> >>> Daniel >>> >>> >>> >>> On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users > wrote: >>>> I think it should work with static libraries and 64bit compilers. >>>> >>>> That's how I think --download-f2cblaslapack [etc] work. >>>> >>>> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. >>>> >>>> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 >>>> $ nm -Ao msmpi.lib |grep " MPI_Init" >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized >>>> >>>> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. >>>> >>>> Checking PETSc dll build: >>>> >>>> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib >>>> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays >>>> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays >>>> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays >>>> >>>> It also has the unmangled symbol - so I guess this mode can work generally with dlls. >>>> >>>> Satish >>>> >>>> >>>> On Wed, 19 Jul 2023, Barry Smith wrote: >>>> >>>>> >>>>> Satish, >>>>> >>>>> So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: >>>>>> >>>>>> BTW: Some explanation of configure: >>>>>> >>>>>> It attempts the following on linux: >>>>>> >>>>>>>>>>>> >>>>>> Source: >>>>>> #include "confdefs.h" >>>>>> #include "conffix.h" >>>>>> /* Override any gcc2 internal prototype to avoid an error. */ >>>>>> char HYPRE_IJMatrixCreate(); >>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>> >>>>>> int main(void) { >>>>>> _check_HYPRE_IJMatrixCreate(); >>>>>> return 0; >>>>>> } >>>>>> <<<<<<< >>>>>> >>>>>> Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); >>>>>> >>>>>> Compiling it manually: >>>>>> >>>>>>>>>> >>>>>> [balay at pj01 petsc]$ cat conftest.c >>>>>> char HYPRE_IJMatrixCreate(); >>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>> >>>>>> int main(void) { >>>>>> _check_HYPRE_IJMatrixCreate(); >>>>>> return 0; >>>>>> } >>>>>> [balay at pj01 petsc]$ gcc -c conftest.c >>>>>> [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate >>>>>> conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate >>>>>> conftest.o: U HYPRE_IJMatrixCreate >>>>>> [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate >>>>>> arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate >>>>>> [balay at pj01 petsc]$ >>>>>> <<<< >>>>>> >>>>>> Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! >>>>>> >>>>>>>>>>>> >>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so >>>>>> [balay at pj01 petsc]$ echo $? >>>>>> 0 >>>>>> <<<<< >>>>>> >>>>>> On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. >>>>>> >>>>>> Satish >>>>>> >>>>>> >>>>>> On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: >>>>>> >>>>>>> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. >>>>>>> >>>>>>> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py >>>>>>> index 5bc88322aa2..2d6c7932e17 100644 >>>>>>> --- a/config/BuildSystem/config/packages/hypre.py >>>>>>> +++ b/config/BuildSystem/config/packages/hypre.py >>>>>>> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): >>>>>>> self.requiresversion = 1 >>>>>>> self.gitcommit = 'v'+self.version >>>>>>> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] >>>>>>> - self.functions = ['HYPRE_IJMatrixCreate'] >>>>>>> + self.functions = [] >>>>>>> self.includes = ['HYPRE.h'] >>>>>>> self.liblist = [['libHYPRE.a']] >>>>>>> self.buildLanguages = ['C','Cxx'] >>>>>>> >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: >>>>>>> >>>>>>>> >>>>>>>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared >>>>>>>> >>>>>>>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. >>>>>>>> >>>>>>>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone > wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works >>>>>>>>> by setting up a VS project to run some of the example programs. >>>>>>>>> >>>>>>>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications >>>>>>>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc >>>>>>>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). >>>>>>>>> >>>>>>>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: >>>>>>>>> >>>>>>>>> #include "confdefs.h" >>>>>>>>> #include "conffix.h" >>>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ >>>>>>>>> >>>>>>>>> #include "HYPRE.h" >>>>>>>>> >>>>>>>>> char HYPRE_IJMatrixCreate(); >>>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>>>>> >>>>>>>>> int main() { >>>>>>>>> _check_HYPRE_IJMatrixCreate();; >>>>>>>>> return 0; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that >>>>>>>>> include PTScotch. >>>>>>>>> >>>>>>>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: >>>>>>>>> >>>>>>>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... >>>>>>>>> >>>>>>>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more >>>>>>>>> nontrivial Hypre example programs, I get the same error: >>>>>>>>> >>>>>>>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 >>>>>>>>> >>>>>>>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but >>>>>>>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. >>>>>>>>> >>>>>>>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? >>>>>>>>> >>>>>>>>> Many thanks, >>>>>>>>> >>>>>>>>> Daniel >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >> From Robert.Crockett at lamresearch.com Fri Jul 21 10:11:21 2023 From: Robert.Crockett at lamresearch.com (Robert.Crockett at lamresearch.com) Date: Fri, 21 Jul 2023 15:11:21 +0000 Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 Message-ID: Hello, I built PETSc with -with-cxx=0 in order to get around a likely Intel C++ compiler bug. However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using -with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. Robert Crockett Plasma Simulation Engineer | OCTO - Computational Products P: 617.648.8349 M: 415.205.4567 LAM RESEARCH 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com [cid:image001.png at 01D9BBC4.11B5C6D0] LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. Confidential - Limited Access and Use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1167 bytes Desc: image001.png URL: From bsmith at petsc.dev Fri Jul 21 10:35:05 2023 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 21 Jul 2023 11:35:05 -0400 Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: References: Message-ID: <87AAB0CC-1A67-4F7F-99ED-B444BC6001E6@petsc.dev> No, you will need to build MPICH yourself, stand-alone and then direct PETSc's configure to use what you have built. Barry > On Jul 21, 2023, at 11:11 AM, Robert Crockett via petsc-users wrote: > > Hello, > I built PETSc with ?with-cxx=0 in order to get around a likely Intel C++ compiler bug. > However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. > > Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using ?with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. > Robert Crockett > Plasma Simulation Engineer | OCTO - Computational Products > P: 617.648.8349 M: 415.205.4567 > > LAM RESEARCH > 4650 Cushing Pkwy, Fremont CA 94538 USA > lamresearch.com > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > Confidential ? Limited Access and Use > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Fri Jul 21 10:17:51 2023 From: pierre at joliv.et (Pierre Jolivet) Date: Fri, 21 Jul 2023 17:17:51 +0200 Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: References: Message-ID: <89EB5052-0F42-4411-8D6C-C949BEE67E72@joliv.et> > On 21 Jul 2023, at 5:11 PM, Robert Crockett via petsc-users wrote: > > Hello, > I built PETSc with ?with-cxx=0 in order to get around a likely Intel C++ compiler bug. > However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. > > Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using ?with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. You could use $ export MPICH_CXX=icpc But the fact that there is a mismatch is not a great sign. Thanks, Pierre > Robert Crockett > Plasma Simulation Engineer | OCTO - Computational Products > P: 617.648.8349 M: 415.205.4567 > > LAM RESEARCH > 4650 Cushing Pkwy, Fremont CA 94538 USA > lamresearch.com > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > Confidential ? Limited Access and Use > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jul 21 11:49:52 2023 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 21 Jul 2023 12:49:52 -0400 Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: References: <87AAB0CC-1A67-4F7F-99ED-B444BC6001E6@petsc.dev> Message-ID: <69A9AFC2-FCFD-4EAE-8693-D0EFBC6D9098@petsc.dev> You need to look in the configure.log you to see the exact configure/cmake command PETSc configure is using for each package it builds specific to that run of PETSc configure. We do not save them in some other place. > On Jul 21, 2023, at 12:14 PM, Robert.Crockett at lamresearch.com wrote: > > Can I easily get the MPICH config PETSc uses? I?m poking through the repo and not seeing anything related to config of downloaded packages. > Thanks, > Robert > > From: Barry Smith > > Sent: Friday, July 21, 2023 11:35 AM > To: Crockett, Robert > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 > > You don't often get email from bsmith at petsc.dev . Learn why this is important > > External Email: Do NOT reply, click on links, or open attachments unless you recognize the sender and know the content is safe. If you believe this email may be unsafe, please click on the ?Report Phishing? button on the top right of Outlook. > > > > > No, you will need to build MPICH yourself, stand-alone and then direct PETSc's configure to use what you have built. > > Barry > > > > On Jul 21, 2023, at 11:11 AM, Robert Crockett via petsc-users > wrote: > > Hello, > I built PETSc with ?with-cxx=0 in order to get around a likely Intel C++ compiler bug. > However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. > > Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using ?with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. > Robert Crockett > Plasma Simulation Engineer | OCTO - Computational Products > P: 617.648.8349 M: 415.205.4567 > > LAM RESEARCH > 4650 Cushing Pkwy, Fremont CA 94538 USA > lamresearch.com > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > > Confidential ? Limited Access and Use > > > Confidential ? Limited Access and Use > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jul 21 11:57:09 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 21 Jul 2023 11:57:09 -0500 (CDT) Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: <69A9AFC2-FCFD-4EAE-8693-D0EFBC6D9098@petsc.dev> References: <87AAB0CC-1A67-4F7F-99ED-B444BC6001E6@petsc.dev> <69A9AFC2-FCFD-4EAE-8693-D0EFBC6D9098@petsc.dev> Message-ID: <13184233-5623-8afb-1731-414c5b9f0d19@mcs.anl.gov> Were you able to try Jacob's fix - so you could build with cxx? Wrt building external pkgs - one way: - first build pkgs: ./configure PETSC_ARCH=arch-pkgs --prefix=$HOME/soft/petsc-pkgs --with-cc=icc --with-cxx=icpc --with-fc=ifort --download-mpich --download-suitesparse - now build PETSc with these pkgs ./configure PETSC_ARCH=arch-mybuild --with-mpi-dir=$HOME/soft/petsc-pkgs --with-cxx=0 [can't use --with-suitesparse-dir here - due to cxx dependencies within configure - but you disable this dependency check in configure - by commenting out one line of code] Wrt configure options used - they are listed in configure.log - also in - for ex: [balay at pj01 petsc]$ ls arch-linux-c-debug/externalpackages/*/*petsc* arch-linux-c-debug/externalpackages/git.sowing/sowing.petscconf Satish On Fri, 21 Jul 2023, Barry Smith wrote: > > You need to look in the configure.log you to see the exact configure/cmake command PETSc configure is using for each package it builds specific to that run of PETSc configure. We do not save them in some other place. > > > > > On Jul 21, 2023, at 12:14 PM, Robert.Crockett at lamresearch.com wrote: > > > > Can I easily get the MPICH config PETSc uses? I?m poking through the repo and not seeing anything related to config of downloaded packages. > > Thanks, > > Robert > > > > From: Barry Smith > > > Sent: Friday, July 21, 2023 11:35 AM > > To: Crockett, Robert > > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 > > > > You don't often get email from bsmith at petsc.dev . Learn why this is important > > > > External Email: Do NOT reply, click on links, or open attachments unless you recognize the sender and know the content is safe. If you believe this email may be unsafe, please click on the ?Report Phishing? button on the top right of Outlook. > > > > > > > > > > No, you will need to build MPICH yourself, stand-alone and then direct PETSc's configure to use what you have built. > > > > Barry > > > > > > > > On Jul 21, 2023, at 11:11 AM, Robert Crockett via petsc-users > wrote: > > > > Hello, > > I built PETSc with ?with-cxx=0 in order to get around a likely Intel C++ compiler bug. > > However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. > > > > Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using ?with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. > > Robert Crockett > > Plasma Simulation Engineer | OCTO - Computational Products > > P: 617.648.8349 M: 415.205.4567 > > > > LAM RESEARCH > > 4650 Cushing Pkwy, Fremont CA 94538 USA > > lamresearch.com > > > > > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > > > > > Confidential ? Limited Access and Use > > > > > > Confidential ? Limited Access and Use > > > > From balay at mcs.anl.gov Fri Jul 21 11:58:26 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 21 Jul 2023 11:58:26 -0500 (CDT) Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: <13184233-5623-8afb-1731-414c5b9f0d19@mcs.anl.gov> References: <87AAB0CC-1A67-4F7F-99ED-B444BC6001E6@petsc.dev> <69A9AFC2-FCFD-4EAE-8693-D0EFBC6D9098@petsc.dev> <13184233-5623-8afb-1731-414c5b9f0d19@mcs.anl.gov> Message-ID: <3c201549-3983-dce0-4049-68dc175a5c85@mcs.anl.gov> On Fri, 21 Jul 2023, Satish Balay via petsc-users wrote: > Were you able to try Jacob's fix - so you could build with cxx? > > Wrt building external pkgs - one way: > > - first build pkgs: > ./configure PETSC_ARCH=arch-pkgs --prefix=$HOME/soft/petsc-pkgs --with-cc=icc --with-cxx=icpc --with-fc=ifort --download-mpich --download-suitesparse > > - now build PETSc with these pkgs > ./configure PETSC_ARCH=arch-mybuild --with-mpi-dir=$HOME/soft/petsc-pkgs --with-cxx=0 > > [can't use --with-suitesparse-dir here - due to cxx dependencies within configure - but you disable this dependency check in configure - by commenting out one line of code] > > Wrt configure options used - they are listed in configure.log - also in - for ex: > > [balay at pj01 petsc]$ ls arch-linux-c-debug/externalpackages/*/*petsc* > arch-linux-c-debug/externalpackages/git.sowing/sowing.petscconf Also at: [balay at pj01 petsc]$ ls arch-linux-c-debug/lib/petsc/conf/pkg.conf* arch-linux-c-debug/lib/petsc/conf/pkg.conf.sowing Satish > > Satish > > On Fri, 21 Jul 2023, Barry Smith wrote: > > > > > You need to look in the configure.log you to see the exact configure/cmake command PETSc configure is using for each package it builds specific to that run of PETSc configure. We do not save them in some other place. > > > > > > > > > On Jul 21, 2023, at 12:14 PM, Robert.Crockett at lamresearch.com wrote: > > > > > > Can I easily get the MPICH config PETSc uses? I?m poking through the repo and not seeing anything related to config of downloaded packages. > > > Thanks, > > > Robert > > > > > > From: Barry Smith > > > > Sent: Friday, July 21, 2023 11:35 AM > > > To: Crockett, Robert > > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 > > > > > > You don't often get email from bsmith at petsc.dev . Learn why this is important > > > > > > External Email: Do NOT reply, click on links, or open attachments unless you recognize the sender and know the content is safe. If you believe this email may be unsafe, please click on the ?Report Phishing? button on the top right of Outlook. > > > > > > > > > > > > > > > No, you will need to build MPICH yourself, stand-alone and then direct PETSc's configure to use what you have built. > > > > > > Barry > > > > > > > > > > > > On Jul 21, 2023, at 11:11 AM, Robert Crockett via petsc-users > wrote: > > > > > > Hello, > > > I built PETSc with ?with-cxx=0 in order to get around a likely Intel C++ compiler bug. > > > However, the MPICH that also gets built by PETSc then picks up the wrong C++ compiler; mpicxx -show indicates that it is using G++, while mpicc is correctly using icc. > > > > > > Is there a way to get PETSc to pass the correct C++ compiler for the MPICH build when using ?with-cxx=0? I need to compile parts of my own program with mpicxx/icpc. > > > Robert Crockett > > > Plasma Simulation Engineer | OCTO - Computational Products > > > P: 617.648.8349 M: 415.205.4567 > > > > > > LAM RESEARCH > > > 4650 Cushing Pkwy, Fremont CA 94538 USA > > > lamresearch.com > > > > > > > > > > > > LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, "E-mail Transmission") may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. > > > > > > > > > Confidential ? Limited Access and Use > > > > > > > > > Confidential ? Limited Access and Use > > > > > > > > From balay at mcs.anl.gov Fri Jul 21 12:32:11 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 21 Jul 2023 12:32:11 -0500 (CDT) Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <3D80FBFC-58A1-43E5-AD3D-9D18790796BB@petsc.dev> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> <3D80FBFC-58A1-43E5-AD3D-9D18790796BB@petsc.dev> Message-ID: <33006ce7-4851-6d86-6a7a-c3efe79415a3@mcs.anl.gov> We don't have a good way to verify if the changes continue to work with the minver of the pkg [and if minor fixes can get this working - without increasing this requirement - sure without access to the new features provided by the new version] Having a single version dependency between pkgs makes the whole ecosystem [where many pkgs with all their dependencies mixed in] brittle.. Satish On Thu, 20 Jul 2023, Barry Smith wrote: > > Satish, > > I think hypre.py in main needs minversion = 2.29 instead of 2.14 > > > > > On Jul 20, 2023, at 11:05 AM, Satish Balay wrote: > > > > Can check config/BuildSystem/config/packages/hypre.py > > > > petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc 'main' branch with 2.29.0 > > > > Satish > > > > On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote: > > > >> > >> You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed. > >> > >> Barry > >> > >> > >>> On Jul 20, 2023, at 5:10 AM, Daniel Stone wrote: > >>> > >>> Hi All, > >>> > >>> Many thanks for the detailed explainations and ideas! > >>> > >>> I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: > >>> > >>> -------------------------------- > >>> > >>> FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o > >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o > >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o > >>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o > >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' > >>> hypre__global_error = 0; > >>> ^ ~ > >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > >>> __VA_ARGS__; \ > >>> ^~~~~~~~~~~ > >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' > >>> hypre__global_error = 0; > >>> ^ ~ > >>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' > >>> __VA_ARGS__; \ > >>> ^~~~~~~~~~~ > >>> 2 errors generated. > >>> make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 > >>> make[3]: *** Waiting for unfinished jobs.... > >>> FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o > >>> CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o > >>> CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o > >>> make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > >>> make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 > >>> make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > >>> **************************ERROR************************************* > >>> Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log > >>> Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > >>> ******************************************************************** > >>> Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 > >>> > >>> ---------------------------------------- > >>> > >>> But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" > >>> cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied > >>> into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. > >>> > >>> > >>> On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, > >>> which I didn't change: > >>> > >>> # Configuration options > >>> option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) > >>> option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) > >>> option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) > >>> [....] > >>> > >>> > >>> checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. > >>> > >>> > >>> > >>> Thanks again, > >>> > >>> Daniel > >>> > >>> > >>> > >>> On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users > wrote: > >>>> I think it should work with static libraries and 64bit compilers. > >>>> > >>>> That's how I think --download-f2cblaslapack [etc] work. > >>>> > >>>> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. > >>>> > >>>> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 > >>>> $ nm -Ao msmpi.lib |grep " MPI_Init" > >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > >>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > >>>> > >>>> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. > >>>> > >>>> Checking PETSc dll build: > >>>> > >>>> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > >>>> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > >>>> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays > >>>> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays > >>>> > >>>> It also has the unmangled symbol - so I guess this mode can work generally with dlls. > >>>> > >>>> Satish > >>>> > >>>> > >>>> On Wed, 19 Jul 2023, Barry Smith wrote: > >>>> > >>>>> > >>>>> Satish, > >>>>> > >>>>> So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). > >>>>> > >>>>> Barry > >>>>> > >>>>> > >>>>>> On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: > >>>>>> > >>>>>> BTW: Some explanation of configure: > >>>>>> > >>>>>> It attempts the following on linux: > >>>>>> > >>>>>>>>>>>> > >>>>>> Source: > >>>>>> #include "confdefs.h" > >>>>>> #include "conffix.h" > >>>>>> /* Override any gcc2 internal prototype to avoid an error. */ > >>>>>> char HYPRE_IJMatrixCreate(); > >>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >>>>>> > >>>>>> int main(void) { > >>>>>> _check_HYPRE_IJMatrixCreate(); > >>>>>> return 0; > >>>>>> } > >>>>>> <<<<<<< > >>>>>> > >>>>>> Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); > >>>>>> > >>>>>> Compiling it manually: > >>>>>> > >>>>>>>>>> > >>>>>> [balay at pj01 petsc]$ cat conftest.c > >>>>>> char HYPRE_IJMatrixCreate(); > >>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >>>>>> > >>>>>> int main(void) { > >>>>>> _check_HYPRE_IJMatrixCreate(); > >>>>>> return 0; > >>>>>> } > >>>>>> [balay at pj01 petsc]$ gcc -c conftest.c > >>>>>> [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > >>>>>> conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > >>>>>> conftest.o: U HYPRE_IJMatrixCreate > >>>>>> [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate > >>>>>> arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate > >>>>>> [balay at pj01 petsc]$ > >>>>>> <<<< > >>>>>> > >>>>>> Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! > >>>>>> > >>>>>>>>>>>> > >>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so > >>>>>> [balay at pj01 petsc]$ echo $? > >>>>>> 0 > >>>>>> <<<<< > >>>>>> > >>>>>> On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. > >>>>>> > >>>>>> Satish > >>>>>> > >>>>>> > >>>>>> On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > >>>>>> > >>>>>>> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. > >>>>>>> > >>>>>>> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py > >>>>>>> index 5bc88322aa2..2d6c7932e17 100644 > >>>>>>> --- a/config/BuildSystem/config/packages/hypre.py > >>>>>>> +++ b/config/BuildSystem/config/packages/hypre.py > >>>>>>> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > >>>>>>> self.requiresversion = 1 > >>>>>>> self.gitcommit = 'v'+self.version > >>>>>>> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > >>>>>>> - self.functions = ['HYPRE_IJMatrixCreate'] > >>>>>>> + self.functions = [] > >>>>>>> self.includes = ['HYPRE.h'] > >>>>>>> self.liblist = [['libHYPRE.a']] > >>>>>>> self.buildLanguages = ['C','Cxx'] > >>>>>>> > >>>>>>> > >>>>>>> Satish > >>>>>>> > >>>>>>> > >>>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared > >>>>>>>> > >>>>>>>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. > >>>>>>>> > >>>>>>>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. > >>>>>>>> > >>>>>>>> Barry > >>>>>>>> > >>>>>>>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone > wrote: > >>>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works > >>>>>>>>> by setting up a VS project to run some of the example programs. > >>>>>>>>> > >>>>>>>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications > >>>>>>>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc > >>>>>>>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). > >>>>>>>>> > >>>>>>>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: > >>>>>>>>> > >>>>>>>>> #include "confdefs.h" > >>>>>>>>> #include "conffix.h" > >>>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ > >>>>>>>>> > >>>>>>>>> #include "HYPRE.h" > >>>>>>>>> > >>>>>>>>> char HYPRE_IJMatrixCreate(); > >>>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } > >>>>>>>>> > >>>>>>>>> int main() { > >>>>>>>>> _check_HYPRE_IJMatrixCreate();; > >>>>>>>>> return 0; > >>>>>>>>> } > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that > >>>>>>>>> include PTScotch. > >>>>>>>>> > >>>>>>>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: > >>>>>>>>> > >>>>>>>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... > >>>>>>>>> > >>>>>>>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more > >>>>>>>>> nontrivial Hypre example programs, I get the same error: > >>>>>>>>> > >>>>>>>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > >>>>>>>>> > >>>>>>>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but > >>>>>>>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. > >>>>>>>>> > >>>>>>>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? > >>>>>>>>> > >>>>>>>>> Many thanks, > >>>>>>>>> > >>>>>>>>> Daniel > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > >> > From Robert.Crockett at lamresearch.com Fri Jul 21 13:24:39 2023 From: Robert.Crockett at lamresearch.com (Robert.Crockett at lamresearch.com) Date: Fri, 21 Jul 2023 18:24:39 +0000 Subject: [petsc-users] MPICH C++ compilers when using PETSC --with-cxx=0 In-Reply-To: <13184233-5623-8afb-1731-414c5b9f0d19@mcs.anl.gov> References: <87AAB0CC-1A67-4F7F-99ED-B444BC6001E6@petsc.dev> <69A9AFC2-FCFD-4EAE-8693-D0EFBC6D9098@petsc.dev> <13184233-5623-8afb-1731-414c5b9f0d19@mcs.anl.gov> Message-ID: No Jacob's fixes have not remedied the errors, unfortunately. Not for trying - he has sent me an update since the first. I'll build MPI separately+ADs- thanks for the tips. +AH4-Robert Confidential +IBM- Limited Access and Use -----Original Message----- From: Satish Balay +ADw-balay+AEA-mcs.anl.gov+AD4- Sent: Friday, July 21, 2023 12:57 PM To: Barry Smith +ADw-bsmith+AEA-petsc.dev+AD4- Cc: Crockett, Robert +ADw-Robert.Crockett+AEA-lamresearch.com+AD4AOw- petsc-users +ADw-petsc-users+AEA-mcs.anl.gov+AD4- Subject: Re: +AFs-petsc-users+AF0- MPICH C+-+- compilers when using PETSC --with-cxx+AD0-0 +AFs-You don't often get email from balay+AEA-mcs.anl.gov. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification +AF0- External Email: Do NOT reply, click on links, or open attachments unless you recognize the sender and know the content is safe. If you believe this email may be unsafe, please click on the +IBw-Report Phishing+IB0- button on the top right of Outlook. Were you able to try Jacob's fix - so you could build with cxx? Wrt building external pkgs - one way: - first build pkgs: ./configure PETSC+AF8-ARCH+AD0-arch-pkgs --prefix+AD0AJA-HOME/soft/petsc-pkgs --with-cc+AD0-icc --with-cxx+AD0-icpc --with-fc+AD0-ifort --download-mpich --download-suitesparse - now build PETSc with these pkgs ./configure PETSC+AF8-ARCH+AD0-arch-mybuild --with-mpi-dir+AD0AJA-HOME/soft/petsc-pkgs --with-cxx+AD0-0 +AFs-can't use --with-suitesparse-dir here - due to cxx dependencies within configure - but you disable this dependency check in configure - by commenting out one line of code+AF0- Wrt configure options used - they are listed in configure.log - also in - for ex: +AFs-balay+AEA-pj01 petsc+AF0AJA- ls arch-linux-c-debug/externalpackages/+ACo-/+ACo-petsc+ACo- arch-linux-c-debug/externalpackages/git.sowing/sowing.petscconf Satish On Fri, 21 Jul 2023, Barry Smith wrote: +AD4- +AD4- You need to look in the configure.log you to see the exact configure/cmake command PETSc configure is using for each package it builds specific to that run of PETSc configure. We do not save them in some other place. +AD4- +AD4- +AD4- +AD4- +AD4- On Jul 21, 2023, at 12:14 PM, Robert.Crockett+AEA-lamresearch.com wrote: +AD4- +AD4- +AD4- +AD4- Can I easily get the MPICH config PETSc uses? I+IBk-m poking through the repo and not seeing anything related to config of downloaded packages. +AD4- +AD4- Thanks, +AD4- +AD4- Robert +AD4- +AD4- +AD4- +AD4- From: Barry Smith +ADw-bsmith+AEA-petsc.dev +ADw-mailto:bsmith+AEA-petsc.dev+AD4APg- +AD4- +AD4- Sent: Friday, July 21, 2023 11:35 AM +AD4- +AD4- To: Crockett, Robert +ADw-Robert.Crockett+AEA-lamresearch.com +AD4- +AD4- +ADw-mailto:Robert.Crockett+AEA-lamresearch.com+AD4APg- +AD4- +AD4- Cc: petsc-users+AEA-mcs.anl.gov +ADw-mailto:petsc-users+AEA-mcs.anl.gov+AD4- +AD4- +AD4- Subject: Re: +AFs-petsc-users+AF0- MPICH C+-+- compilers when using PETSC +AD4- +AD4- --with-cxx+AD0-0 +AD4- +AD4- +AD4- +AD4- You don't often get email from bsmith+AEA-petsc.dev +AD4- +AD4- +ADw-mailto:bsmith+AEA-petsc.dev+AD4-. Learn why this is important +AD4- +AD4- +ADw-https://aka.ms/LearnAboutSenderIdentification+AD4- +AD4- +AD4- +AD4- +AD4- External Email: Do NOT reply, click on links, or open attachments unless you recognize the sender and know the content is safe. If you believe this email may be unsafe, please click on the +IBw-Report Phishing+IB0- button on the top right of Outlook. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- No, you will need to build MPICH yourself, stand-alone and then direct PETSc's configure to use what you have built. +AD4- +AD4- +AD4- +AD4- Barry +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- On Jul 21, 2023, at 11:11 AM, Robert Crockett via petsc-users +ADw-petsc-users+AEA-mcs.anl.gov +ADw-mailto:petsc-users+AEA-mcs.anl.gov+AD4APg- wrote: +AD4- +AD4- +AD4- +AD4- Hello, +AD4- +AD4- I built PETSc with +IBM-with-cxx+AD0-0 in order to get around a likely Intel C+-+- compiler bug. +AD4- +AD4- However, the MPICH that also gets built by PETSc then picks up the wrong C+-+- compiler+ADs- mpicxx -show indicates that it is using G+-+-, while mpicc is correctly using icc. +AD4- +AD4- +AD4- +AD4- Is there a way to get PETSc to pass the correct C+-+- compiler for the MPICH build when using +IBM-with-cxx+AD0-0? I need to compile parts of my own program with mpicxx/icpc. +AD4- +AD4- Robert Crockett +AD4- +AD4- Plasma Simulation Engineer +AHw- OCTO - Computational Products +AD4- +AD4- P: 617.648.8349 M: 415.205.4567 +AD4- +AD4- +AD4- +AD4- LAM RESEARCH +AD4- +AD4- 4650 Cushing Pkwy, Fremont CA 94538 USA lamresearch.com +AD4- +AD4- +ADw-https://www.lamresearch.com/+AD4- +ADw-image001.png+AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- LAM RESEARCH CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files, or previous e-mail messages attached to it, (collectively, +ACI-E-mail Transmission+ACI-) may be subject to one or more of the following based on the associated sensitivity level: E-mail Transmission (i) contains confidential information, (ii) is prohibited from distribution outside of Lam, and/or (iii) is intended solely for and restricted to the specified recipient(s). If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender and destroy the original transmission and its attachments without reading them or saving them to disk. Thank you. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- Confidential +IBM- Limited Access and Use +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- Confidential +IBM- Limited Access and Use +AD4- +AD4- +AD4- +AD4- From facklerpw at ornl.gov Fri Jul 21 14:40:48 2023 From: facklerpw at ornl.gov (Fackler, Philip) Date: Fri, 21 Jul 2023 19:40:48 +0000 Subject: [petsc-users] [EXTERNAL] Re: Initializing kokkos before petsc causes a problem In-Reply-To: References: Message-ID: Thanks Junchao! Your fix worked perfectly. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Fackler, Philip via Xolotl-psi-development Sent: Tuesday, June 27, 2023 11:46 To: Junchao Zhang Cc: petsc-users at mcs.anl.gov ; xolotl-psi-development at lists.sourceforge.net Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem OK, great! I'll try it out soon. Thank you, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang Sent: Tuesday, June 27, 2023 10:58 To: Fackler, Philip Cc: petsc-users at mcs.anl.gov ; Blondel, Sophie ; xolotl-psi-development at lists.sourceforge.net Subject: Re: [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem Hi, Philip, It's my fault. I should follow up early that this problem was fixed by https://gitlab.com/petsc/petsc/-/merge_requests/6586. Could you try petsc/main? Thanks. --Junchao Zhang On Tue, Jun 27, 2023 at 9:30?AM Fackler, Philip > wrote: Good morning Junchao! I'm following up here to see if there is any update to petsc to resolve this issue, or if we need to come up with a work-around. Thank you, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang > Sent: Wednesday, June 7, 2023 22:45 To: Fackler, Philip > Cc: petsc-users at mcs.anl.gov >; Blondel, Sophie >; xolotl-psi-development at lists.sourceforge.net > Subject: [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem Hi, Philip, Thanks for reporting. I will have a look at the issue. --Junchao Zhang On Wed, Jun 7, 2023 at 9:30?AM Fackler, Philip via petsc-users > wrote: I'm encountering a problem in xolotl. We initialize kokkos before initializing petsc. Therefore... The pointer referenced here: https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx#L363 from here: https://gitlab.com/petsc/petsc/-/blob/main/include/petsc_kokkos.hpp remains null because the code to initialize it is skipped here: https://gitlab.com/petsc/petsc/-/blob/main/src/sys/objects/kokkos/kinit.kokkos.cxx#L28 See line 71. Can this be modified to allow for kokkos to have been initialized by the application before initializing petsc? Thank you for your help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jul 21 15:32:23 2023 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 21 Jul 2023 16:32:23 -0400 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <33006ce7-4851-6d86-6a7a-c3efe79415a3@mcs.anl.gov> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> <3D80FBFC-58A1-43E5-AD3D-9D18790796BB@petsc.dev> <33006ce7-4851-6d86-6a7a-c3efe79415a3@mcs.anl.gov> Message-ID: <50D7E299-728F-47A4-B098-C58E3FA6C679@petsc.dev> hypre_Error was changed from an integer to a struct in hypre. PETSc code was changed in main to work with that struct and not work if hypre_Error is an int. This means main cannot work with previous hypre versions (where hypre_Error is an int). Sure, instead of changing the minimum version one could potentially change PETSc's main version source code to work with both old and new hypre, but that would need to be done and has not been done. main is simply broken, it allows configure to succeed with an older version of hypre but PETSc main cannot compile with this, thus confusing developers and users. This does need to be fixed, one way or another. > On Jul 21, 2023, at 1:32 PM, Satish Balay wrote: > > We don't have a good way to verify if the changes continue to work > with the minver of the pkg [and if minor fixes can get this working - > without increasing this requirement - sure without access to the new > features provided by the new version] > > Having a single version dependency between pkgs makes the whole > ecosystem [where many pkgs with all their dependencies mixed in] > brittle.. > > Satish > > On Thu, 20 Jul 2023, Barry Smith wrote: > >> >> Satish, >> >> I think hypre.py in main needs minversion = 2.29 instead of 2.14 >> >> >> >>> On Jul 20, 2023, at 11:05 AM, Satish Balay wrote: >>> >>> Can check config/BuildSystem/config/packages/hypre.py >>> >>> petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc 'main' branch with 2.29.0 >>> >>> Satish >>> >>> On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote: >>> >>>> >>>> You cannot use this version of PETSc, 3.19, with the version of hypre you installed. In hypre they recently changed hypre_Error from an integer to a struct which completely breaks compatibility with previous versions of hypre (and hence previous versions of PETSc). You must use the main git branch of PETSc with the version of hypre you installed. >>>> >>>> Barry >>>> >>>> >>>>> On Jul 20, 2023, at 5:10 AM, Daniel Stone wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Many thanks for the detailed explainations and ideas! >>>>> >>>>> I tried skipping the test. When it came time to do the build itself (make $PETSC_DIR... all) I get some failures, unsurprisingly: >>>>> >>>>> -------------------------------- >>>>> >>>>> FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o >>>>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o >>>>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o >>>>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): error: assigning to 'hypre_Error' from incompatible type 'int' >>>>> hypre__global_error = 0; >>>>> ^ ~ >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' >>>>> __VA_ARGS__; \ >>>>> ^~~~~~~~~~~ >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): error: assigning to 'hypre_Error' from incompatible type 'int' >>>>> hypre__global_error = 0; >>>>> ^ ~ >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): note: expanded from macro 'PetscStackCallExternalVoid' >>>>> __VA_ARGS__; \ >>>>> ^~~~~~~~~~~ >>>>> 2 errors generated. >>>>> make[3]: *** [gmakefile:195: arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 >>>>> make[3]: *** Waiting for unfinished jobs.... >>>>> FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o >>>>> CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o >>>>> CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o >>>>> make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' >>>>> make[2]: *** [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 >>>>> make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' >>>>> **************************ERROR************************************* >>>>> Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log >>>>> Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov >>>>> ******************************************************************** >>>>> Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 >>>>> >>>>> ---------------------------------------- >>>>> >>>>> But wait - isn't this the compile stage, not the linking stage? This seems to imply that I've made a hash of providing include file such that a definition of "hypre_Error" >>>>> cannot be seen - unless I'm misinterpreting. Interesting note about Hypre and include files - if built using configure and make, all the include files are conviniently copied >>>>> into hypre/src/hypre/include. This is not done for a cmake build - I had to do the copying myself. Maybe I missed one. >>>>> >>>>> >>>>> On shared vs. static - if there a clear way of telling which I've ended up with? I've checked the cmakelists for hypre and this seems to imply that not-shared is the default, >>>>> which I didn't change: >>>>> >>>>> # Configuration options >>>>> option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) >>>>> option(HYPRE_ENABLE_BIGINT "Use long long int for HYPRE_Int" OFF) >>>>> option(HYPRE_ENABLE_MIXEDINT "Use long long int for HYPRE_BigInt, int for HYPRE_INT" OFF) >>>>> [....] >>>>> >>>>> >>>>> checking again, I've noticed that the way that the stub-test fails is different depending on whether it's called from the config script or used in isolation - more details on that soon. >>>>> >>>>> >>>>> >>>>> Thanks again, >>>>> >>>>> Daniel >>>>> >>>>> >>>>> >>>>> On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users > wrote: >>>>>> I think it should work with static libraries and 64bit compilers. >>>>>> >>>>>> That's how I think --download-f2cblaslapack [etc] work. >>>>>> >>>>>> Also it works with MS-MPI - even-though its a dll install, the library stub provides this symbol somehow.. >>>>>> >>>>>> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64 >>>>>> $ nm -Ao msmpi.lib |grep " MPI_Init" >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized >>>>>> >>>>>> However - if the library symbol is somehow mangled - this configure mode of checking library functions will fail. >>>>>> >>>>>> Checking PETSc dll build: >>>>>> >>>>>> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib >>>>>> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays >>>>>> libpetsc.lib:libpetsc.dll:0000000000000000 I __imp_MatCreateSeqAIJWithArrays >>>>>> libpetsc.lib:libpetsc.dll:0000000000000000 T MatCreateSeqAIJWithArrays >>>>>> >>>>>> It also has the unmangled symbol - so I guess this mode can work generally with dlls. >>>>>> >>>>>> Satish >>>>>> >>>>>> >>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> Satish, >>>>>>> >>>>>>> So it will always fail on Windows with Windows compilers (both with static and shared libraries)? Is this true for all PETSc external packages? If so, why does the installation documentation say that some external packages can work with Windows compilers? (Presumably PETSc cannot since the configure tests will fail). >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>>> On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: >>>>>>>> >>>>>>>> BTW: Some explanation of configure: >>>>>>>> >>>>>>>> It attempts the following on linux: >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Source: >>>>>>>> #include "confdefs.h" >>>>>>>> #include "conffix.h" >>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ >>>>>>>> char HYPRE_IJMatrixCreate(); >>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>>>> >>>>>>>> int main(void) { >>>>>>>> _check_HYPRE_IJMatrixCreate(); >>>>>>>> return 0; >>>>>>>> } >>>>>>>> <<<<<<< >>>>>>>> >>>>>>>> Note - it does not include 'HYPRE.h' here - but redefines the prototype as 'char HYPRE_IJMatrixCreate(); >>>>>>>> >>>>>>>> Compiling it manually: >>>>>>>> >>>>>>>>>>>> >>>>>>>> [balay at pj01 petsc]$ cat conftest.c >>>>>>>> char HYPRE_IJMatrixCreate(); >>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>>>> >>>>>>>> int main(void) { >>>>>>>> _check_HYPRE_IJMatrixCreate(); >>>>>>>> return 0; >>>>>>>> } >>>>>>>> [balay at pj01 petsc]$ gcc -c conftest.c >>>>>>>> [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate >>>>>>>> conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate >>>>>>>> conftest.o: U HYPRE_IJMatrixCreate >>>>>>>> [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so |grep HYPRE_IJMatrixCreate >>>>>>>> arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T HYPRE_IJMatrixCreate >>>>>>>> [balay at pj01 petsc]$ >>>>>>>> <<<< >>>>>>>> >>>>>>>> Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure succeeds! >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o arch-linux-c-debug/lib/libHYPRE.so >>>>>>>> [balay at pj01 petsc]$ echo $? >>>>>>>> 0 >>>>>>>> <<<<< >>>>>>>> >>>>>>>> On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) etc..] - this might not match - resulting in link failures. >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: >>>>>>>> >>>>>>>>> You could try skipping this test [and assume --with-hypre-include and --with-hypre-lib options are correct] - and see if this works. >>>>>>>>> >>>>>>>>> diff --git a/config/BuildSystem/config/packages/hypre.py b/config/BuildSystem/config/packages/hypre.py >>>>>>>>> index 5bc88322aa2..2d6c7932e17 100644 >>>>>>>>> --- a/config/BuildSystem/config/packages/hypre.py >>>>>>>>> +++ b/config/BuildSystem/config/packages/hypre.py >>>>>>>>> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): >>>>>>>>> self.requiresversion = 1 >>>>>>>>> self.gitcommit = 'v'+self.version >>>>>>>>> self.download = ['git://https://github.com/hypre-space/hypre','https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] >>>>>>>>> - self.functions = ['HYPRE_IJMatrixCreate'] >>>>>>>>> + self.functions = [] >>>>>>>>> self.includes = ['HYPRE.h'] >>>>>>>>> self.liblist = [['libHYPRE.a']] >>>>>>>>> self.buildLanguages = ['C','Cxx'] >>>>>>>>> >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> You don't indicate what type of libraries you built hypre with; static or shared. My guess is you ended up with shared >>>>>>>>>> >>>>>>>>>> I think the answer to your difficulty is hidden in __cdecl (Satish will know much better than me). When you are looking for symbols in Windows shared libraries you have to prepend something to the function prototype to have it successfully found. For example the PETSc include files have these things __declspec(dllimport) The configure test fails because it does not provide the needed prototype. Likely you built PTScotch with static libraries so no problem. >>>>>>>>>> >>>>>>>>>> The simplest fix would be to build static hypre libraries. I think it is a major project to get PETSc configure and macro system to work properly with external packages that are in Windows shared libraries since more use of __declspec would be needed. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> The PETSc installation instructions should probably say something about external packages with Windows shared libraries. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone > wrote: >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I'm working on getting a petsc build running on windows. One necessary package to include is Hypre. I've been able to build Hypre seperately using cmake, and confirmed that the library works >>>>>>>>>>> by setting up a VS project to run some of the example programs. >>>>>>>>>>> >>>>>>>>>>> My attempted petsc build is being done through cygwin. I've been able to (with varying degrees of difficulty), build a fairly plain petsc, and one that downloads and builds ptscotch (after some modifications >>>>>>>>>>> to both ptscotch and the config script). I am now attempting to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). Note that the same compilers are being used for both Hypre and for petsc >>>>>>>>>>> through cygwin - the new intel oneapi compilers (icx and ifx, after again varying amounts of pain to work around their awkwardness with the config script). >>>>>>>>>>> >>>>>>>>>>> I'm seeing a problem when the config script does some tests on the included hypre lib. The source code looks like: >>>>>>>>>>> >>>>>>>>>>> #include "confdefs.h" >>>>>>>>>>> #include "conffix.h" >>>>>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ >>>>>>>>>>> >>>>>>>>>>> #include "HYPRE.h" >>>>>>>>>>> >>>>>>>>>>> char HYPRE_IJMatrixCreate(); >>>>>>>>>>> static void _check_HYPRE_IJMatrixCreate() { HYPRE_IJMatrixCreate(); } >>>>>>>>>>> >>>>>>>>>>> int main() { >>>>>>>>>>> _check_HYPRE_IJMatrixCreate();; >>>>>>>>>>> return 0; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As I understand this is a fairly standard type of stub program used by the config script to check that it is able to link to certain symbols in given libraries. Tests like this have succeeded in my builds that >>>>>>>>>>> include PTScotch. >>>>>>>>>>> >>>>>>>>>>> I keep getting a linker error with the above test, including if I seperate it out and try to build it seperately: >>>>>>>>>>> >>>>>>>>>>> unresolved external symbol "char __cdel HYPRE_IJMatrixCreate(void)" .... >>>>>>>>>>> >>>>>>>>>>> Ok, it looks like a problem with either the library or linker commands. But here's the interesting thing - If I transplant this code into VS, with the same project setting that allows it to build the much more >>>>>>>>>>> nontrivial Hypre example programs, I get the same error: >>>>>>>>>>> >>>>>>>>>>> Error LNK2001 unresolved external symbol "char __cdecl HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 >>>>>>>>>>> >>>>>>>>>>> So it seems like there might be something about this type of stub program that is not working with my Hypre library. I don't fully understand this program - it's able to call the function with no arguments, but >>>>>>>>>>> it also needs to be linked against a library containing the function, apparently by wrapping it in a static void function? Not something I've seen before. >>>>>>>>>>> >>>>>>>>>>> Does anyone have any insight into what might be going wrong - or really just any explaination of how the stub program works so I can figure out why it isn't in this case? >>>>>>>>>>> >>>>>>>>>>> Many thanks, >>>>>>>>>>> >>>>>>>>>>> Daniel >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >> From spradeepmahadeek at gmail.com Sat Jul 22 09:50:01 2023 From: spradeepmahadeek at gmail.com (s.pradeep kumar) Date: Sat, 22 Jul 2023 09:50:01 -0500 Subject: [petsc-users] HELP: How to compile Fortran 2008 files using Petsc. Message-ID: Hi, Petsc by default seems to only recognize only the following files // .SUFFIXES: .F .F90 .f90 ${SUFFIXES} .PETSc .C .cc .cpp .cxx .r .rm .so .html .ad .m .F95 .f95 .fiat .cu .hip.cpp .kokkos.cxx .raja.cxx *.sycl.cxx // I want to compile .F08 files. How should I go about this? Regards, Pradeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jul 22 10:10:43 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 22 Jul 2023 11:10:43 -0400 Subject: [petsc-users] HELP: How to compile Fortran 2008 files using Petsc. In-Reply-To: References: Message-ID: On Sat, Jul 22, 2023 at 10:50?AM s.pradeep kumar wrote: > Hi, > > Petsc by default seems to only recognize only the following files > > // > .SUFFIXES: .F .F90 .f90 ${SUFFIXES} .PETSc .C .cc .cpp .cxx .r .rm .so > .html .ad .m .F95 .f95 .fiat .cu .hip.cpp .kokkos.cxx .raja.cxx *.sycl.cxx > // > > I want to compile .F08 files. How should I go about this? > The different suffixes are telling us to process them differently. I don't think there is any difference in the processing for F90 and F08, so I would just rename those files .F90. Thanks, Matt > Regards, > Pradeep > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Sun Jul 23 10:50:04 2023 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Mon, 24 Jul 2023 00:50:04 +0900 Subject: [petsc-users] FEM Matrix Assembly of Overlapped Mesh Message-ID: Hi, I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) DMPlex and used the MUMPS solver to solve it. But I got different solution by using 1 CPU and MPI parallel computation. I am wondering if I missed some necessary step or setting during my implementation. My calling process like follows 1. Generate matrix pkij from DMPlex dm_mesh ``` PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) ) PetscCallA( MatSetBlockSize(pkij, 1, ierr)) if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr) PetscCallA( MatSetFromOptions(pkij, ierr)) PetscCallA(MatGetLocalSize(pkij, m, n, ierr)); PetscCallA(MatGetSize(pkij, gm, gn, ierr)); PetscCallA(MatGetBlockSize(pkij, bs, ierr)); ! ! Create a preallocator matrix with sizes (local and global) matching the jacobian A. ! PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); PetscCallA(MatSetUp(preallocator, ierr)); ``` 2. Do Matrix preallocation ``` PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); PetscCallA(MatSetUp(preallocator, ierr)); ....... PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) ) PetscCallA( MatDestroy(preallocator,ierr) ) ``` 3. Assemble matrix pkij by calling MatSetValue of all overlapped elements. In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed? Thanks for -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Sun Jul 23 10:53:46 2023 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Mon, 24 Jul 2023 00:53:46 +0900 Subject: [petsc-users] Matrix assembly problem of overlapped DMPlex Message-ID: ?? 0:50 (0 ??) To PETSc Hi, I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) DMPlex and used the MUMPS solver to solve it. But I got a different solution by using 1 CPU and MPI parallel computation. I am wondering if I missed some necessary step or setting during my implementation. My calling process like follows 1. Generate matrix pkij from DMPlex dm_mesh ``` PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) ) PetscCallA( MatSetBlockSize(pkij, 1, ierr)) if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr) PetscCallA( MatSetFromOptions(pkij, ierr)) PetscCallA(MatGetLocalSize(pkij, m, n, ierr)); PetscCallA(MatGetSize(pkij, gm, gn, ierr)); PetscCallA(MatGetBlockSize(pkij, bs, ierr)); ``` 2. Do Matrix preallocation ``` PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); PetscCallA(MatSetUp(preallocator, ierr)); ....... PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) ) PetscCallA( MatDestroy(preallocator,ierr) ) ``` 3. Assemble matrix pkij by calling MatSetValue of all overlapped elements. In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed? Thanks for your help. ------ X. Yuan, Ph.D. of solid mechanics Advancesoft, Japan --------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jul 23 11:41:23 2023 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 23 Jul 2023 12:41:23 -0400 Subject: [petsc-users] FEM Matrix Assembly of Overlapped Mesh In-Reply-To: References: Message-ID: If you want a processor independent solve with mumps use '-pc_type lu' and if you are configured with MUMPS it will give you a parallel LU solve. And don't use any overlap in DM. If want a local LU with global 'asm' or 'bjacobi' then you have an iterative solver and use something like -ksp_type gmres. Mark On Sun, Jul 23, 2023 at 11:50?AM ?? wrote: > Hi, > > I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) > DMPlex and used the MUMPS solver to solve it. But I got different solution > by using 1 CPU and MPI parallel computation. I am wondering if I missed > some necessary step or setting during my implementation. > > My calling process like follows > > 1. Generate matrix pkij from DMPlex dm_mesh > > ``` > PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) ) > PetscCallA( MatSetBlockSize(pkij, 1, ierr)) > if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr) > PetscCallA( MatSetFromOptions(pkij, ierr)) > > PetscCallA(MatGetLocalSize(pkij, m, n, ierr)); > PetscCallA(MatGetSize(pkij, gm, gn, ierr)); > PetscCallA(MatGetBlockSize(pkij, bs, ierr)); > > ! > ! Create a preallocator matrix with sizes (local and global) matching the jacobian A. > ! > PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); > PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); > PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); > PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); > PetscCallA(MatSetUp(preallocator, ierr)); > ``` > > 2. Do Matrix preallocation > > ``` > PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); > PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); > PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); > PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); > PetscCallA(MatSetUp(preallocator, ierr)); > > ....... > > PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) ) > PetscCallA( MatDestroy(preallocator,ierr) ) > ``` > > 3. Assemble matrix pkij by calling MatSetValue of all overlapped elements. > > In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed? > > Thanks for > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 23 15:28:10 2023 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 23 Jul 2023 16:28:10 -0400 Subject: [petsc-users] Matrix assembly problem of overlapped DMPlex In-Reply-To: References: Message-ID: On Sun, Jul 23, 2023 at 11:54?AM ?? wrote: > ?? > 0:50 (0 ??) > To PETSc > Hi, > > I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) > DMPlex and used the MUMPS solver to solve it. But I got a > different solution by using 1 CPU and MPI parallel computation. I am > wondering if I missed some necessary step or setting during my > implementation. > > My calling process like follows > > 1. Generate matrix pkij from DMPlex dm_mesh > > ``` > PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) ) > PetscCallA( MatSetBlockSize(pkij, 1, ierr)) > if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr) > PetscCallA( MatSetFromOptions(pkij, ierr)) > > PetscCallA(MatGetLocalSize(pkij, m, n, ierr)); > PetscCallA(MatGetSize(pkij, gm, gn, ierr)); > PetscCallA(MatGetBlockSize(pkij, bs, ierr)); > ``` > > 2. Do Matrix preallocation > > ``` > PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr)); > PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr)); > PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr)); > PetscCallA(MatSetBlockSize(preallocator, bs, ierr)); > PetscCallA(MatSetUp(preallocator, ierr)); > > ....... > > PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) ) > PetscCallA( MatDestroy(preallocator,ierr) ) > ``` > > 3. Assemble matrix pkij by calling MatSetValue of all overlapped elements. > > In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed? > > Do you want to ignore off-process entries? Usually in FEM we share part of the boundary with other processes, and thus ignoring the off-process entries will not give the correct result. Thanks, Matt > > > Thanks for your help. > > ------ > > X. Yuan, Ph.D. of solid mechanics > > Advancesoft, Japan > > --------- > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.stone at opengosim.com Mon Jul 24 05:21:19 2023 From: daniel.stone at opengosim.com (Daniel Stone) Date: Mon, 24 Jul 2023 11:21:19 +0100 Subject: [petsc-users] Confusion/failures about the tests involved in including Hypre In-Reply-To: <50D7E299-728F-47A4-B098-C58E3FA6C679@petsc.dev> References: <5CF63688-6351-4EA4-9737-7A44A63C061C@petsc.dev> <48e0971e-6211-7d00-dd2e-a965b21a4e22@mcs.anl.gov> <7F01C428-49CE-4E1E-AB72-C2725E585CEB@petsc.dev> <48453d37-48df-6c40-c55f-28bd06fc7486@mcs.anl.gov> <4725AED4-FFFC-4DCD-85CD-6A6E96380523@petsc.dev> <3D80FBFC-58A1-43E5-AD3D-9D18790796BB@petsc.dev> <33006ce7-4851-6d86-6a7a-c3efe79415a3@mcs.anl.gov> <50D7E299-728F-47A4-B098-C58E3FA6C679@petsc.dev> Message-ID: On the hypre versioning - aha. For this project I locked the petsc version a little while ago (3.19.1), but I've been using a fresh clone of hypre, so clearly it's too modern a version. Using the appropriate version of hypre (2.28.0, according to hypre.py) might fix some things. I may have other problems in the form of the confilicting compiler options as Satish mentioned, which I'll have to figure out too. Thanks, Daniel On Fri, Jul 21, 2023 at 9:32?PM Barry Smith wrote: > > > hypre_Error was changed from an integer to a struct in hypre. PETSc code > was changed in main to work with that struct and not work if hypre_Error is > an int. This means main cannot work with previous hypre versions (where > hypre_Error is an int). Sure, instead of changing the minimum version one > could potentially change PETSc's main version source code to work with both > old and new hypre, but that would need to be done and has not been done. > main is simply broken, it allows configure to succeed with an older version > of hypre but PETSc main cannot compile with this, thus confusing developers > and users. This does need to be fixed, one way or another. > > > > > On Jul 21, 2023, at 1:32 PM, Satish Balay wrote: > > > > We don't have a good way to verify if the changes continue to work > > with the minver of the pkg [and if minor fixes can get this working - > > without increasing this requirement - sure without access to the new > > features provided by the new version] > > > > Having a single version dependency between pkgs makes the whole > > ecosystem [where many pkgs with all their dependencies mixed in] > > brittle.. > > > > Satish > > > > On Thu, 20 Jul 2023, Barry Smith wrote: > > > >> > >> Satish, > >> > >> I think hypre.py in main needs minversion = 2.29 instead of 2.14 > >> > >> > >> > >>> On Jul 20, 2023, at 11:05 AM, Satish Balay wrote: > >>> > >>> Can check config/BuildSystem/config/packages/hypre.py > >>> > >>> petsc-3.19 (or release branch) is compatible with hypre 2.28.0, petsc > 'main' branch with 2.29.0 > >>> > >>> Satish > >>> > >>> On Thu, 20 Jul 2023, Barry Smith via petsc-users wrote: > >>> > >>>> > >>>> You cannot use this version of PETSc, 3.19, with the version of hypre > you installed. In hypre they recently changed hypre_Error from an integer > to a struct which completely breaks compatibility with previous versions of > hypre (and hence previous versions of PETSc). You must use the main git > branch of PETSc with the version of hypre you installed. > >>>> > >>>> Barry > >>>> > >>>> > >>>>> On Jul 20, 2023, at 5:10 AM, Daniel Stone < > daniel.stone at opengosim.com> wrote: > >>>>> > >>>>> Hi All, > >>>>> > >>>>> Many thanks for the detailed explainations and ideas! > >>>>> > >>>>> I tried skipping the test. When it came time to do the build itself > (make $PETSC_DIR... all) I get some failures, unsurprisingly: > >>>>> > >>>>> -------------------------------- > >>>>> > >>>>> FC arch-mswin-c-opt/obj/dm/f90-mod/petscdmplexmod.o > >>>>> CC > arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-custom/zhypref.o > >>>>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/ftn-auto/hypref.o > >>>>> CC arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o > >>>>> > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(444,29): > error: assigning to 'hypre_Error' from incompatible type 'int' > >>>>> hypre__global_error = 0; > >>>>> ^ ~ > >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): > note: expanded from macro 'PetscStackCallExternalVoid' > >>>>> __VA_ARGS__; \ > >>>>> ^~~~~~~~~~~ > >>>>> > C:\cygwin64\home\DANIEL~1\PETSC_~1.1\src\ksp\pc\impls\hypre\hypre.c(634,29): > error: assigning to 'hypre_Error' from incompatible type 'int' > >>>>> hypre__global_error = 0; > >>>>> ^ ~ > >>>>> C:\cygwin64\home\DANIEL~1\PETSC_~1.1\include\petscerror.h(1752,7): > note: expanded from macro 'PetscStackCallExternalVoid' > >>>>> __VA_ARGS__; \ > >>>>> ^~~~~~~~~~~ > >>>>> 2 errors generated. > >>>>> make[3]: *** [gmakefile:195: > arch-mswin-c-opt/obj/ksp/pc/impls/hypre/hypre.o] Error 1 > >>>>> make[3]: *** Waiting for unfinished jobs.... > >>>>> FC arch-mswin-c-opt/obj/ksp/f90-mod/petsckspdefmod.o > >>>>> CC arch-mswin-c-opt/obj/dm/impls/da/hypre/mhyp.o > >>>>> CC arch-mswin-c-opt/obj/mat/impls/hypre/mhypre.o > >>>>> make[3]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > >>>>> make[2]: *** > [/home/DanielOGS/petsc_ogs_3.19.1/lib/petsc/conf/rules.doc:28: libs] Error 2 > >>>>> make[2]: Leaving directory '/home/DanielOGS/petsc_ogs_3.19.1' > >>>>> **************************ERROR************************************* > >>>>> Error during compile, check arch-mswin-c-opt/lib/petsc/conf/make.log > >>>>> Send it and arch-mswin-c-opt/lib/petsc/conf/configure.log to > petsc-maint at mcs.anl.gov > >>>>> ******************************************************************** > >>>>> Finishing make run at Wed, 19 Jul 2023 17:07:00 +0100 > >>>>> > >>>>> ---------------------------------------- > >>>>> > >>>>> But wait - isn't this the compile stage, not the linking stage? This > seems to imply that I've made a hash of providing include file such that a > definition of "hypre_Error" > >>>>> cannot be seen - unless I'm misinterpreting. Interesting note about > Hypre and include files - if built using configure and make, all the > include files are conviniently copied > >>>>> into hypre/src/hypre/include. This is not done for a cmake build - I > had to do the copying myself. Maybe I missed one. > >>>>> > >>>>> > >>>>> On shared vs. static - if there a clear way of telling which I've > ended up with? I've checked the cmakelists for hypre and this seems to > imply that not-shared is the default, > >>>>> which I didn't change: > >>>>> > >>>>> # Configuration options > >>>>> option(HYPRE_ENABLE_SHARED "Build a shared library" OFF) > >>>>> option(HYPRE_ENABLE_BIGINT "Use long long int for > HYPRE_Int" OFF) > >>>>> option(HYPRE_ENABLE_MIXEDINT "Use long long int for > HYPRE_BigInt, int for HYPRE_INT" OFF) > >>>>> [....] > >>>>> > >>>>> > >>>>> checking again, I've noticed that the way that the stub-test fails > is different depending on whether it's called from the config script or > used in isolation - more details on that soon. > >>>>> > >>>>> > >>>>> > >>>>> Thanks again, > >>>>> > >>>>> Daniel > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Jul 19, 2023 at 4:58?PM Satish Balay via petsc-users < > petsc-users at mcs.anl.gov > wrote: > >>>>>> I think it should work with static libraries and 64bit compilers. > >>>>>> > >>>>>> That's how I think --download-f2cblaslapack [etc] work. > >>>>>> > >>>>>> Also it works with MS-MPI - even-though its a dll install, the > library stub provides this symbol somehow.. > >>>>>> > >>>>>> balay at ps5 /cygdrive/c/Program Files (x86)/Microsoft > SDKs/MPI/Lib/x64 > >>>>>> $ nm -Ao msmpi.lib |grep " MPI_Init" > >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init > >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Init_thread > >>>>>> msmpi.lib:msmpi.dll:0000000000000000 T MPI_Initialized > >>>>>> > >>>>>> However - if the library symbol is somehow mangled - this configure > mode of checking library functions will fail. > >>>>>> > >>>>>> Checking PETSc dll build: > >>>>>> > >>>>>> balay at ps5 ~/petsc/arch-ci-mswin-uni/lib > >>>>>> $ nm -Ao libpetsc.lib |grep MatCreateSeqAIJWithArrays > >>>>>> libpetsc.lib:libpetsc.dll:0000000000000000 I > __imp_MatCreateSeqAIJWithArrays > >>>>>> libpetsc.lib:libpetsc.dll:0000000000000000 T > MatCreateSeqAIJWithArrays > >>>>>> > >>>>>> It also has the unmangled symbol - so I guess this mode can work > generally with dlls. > >>>>>> > >>>>>> Satish > >>>>>> > >>>>>> > >>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: > >>>>>> > >>>>>>> > >>>>>>> Satish, > >>>>>>> > >>>>>>> So it will always fail on Windows with Windows compilers (both > with static and shared libraries)? Is this true for all PETSc external > packages? If so, why does the installation documentation say that some > external packages can work with Windows compilers? (Presumably PETSc cannot > since the configure tests will fail). > >>>>>>> > >>>>>>> Barry > >>>>>>> > >>>>>>> > >>>>>>>> On Jul 19, 2023, at 11:40 AM, Satish Balay > wrote: > >>>>>>>> > >>>>>>>> BTW: Some explanation of configure: > >>>>>>>> > >>>>>>>> It attempts the following on linux: > >>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>> Source: > >>>>>>>> #include "confdefs.h" > >>>>>>>> #include "conffix.h" > >>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ > >>>>>>>> char HYPRE_IJMatrixCreate(); > >>>>>>>> static void _check_HYPRE_IJMatrixCreate() { > HYPRE_IJMatrixCreate(); } > >>>>>>>> > >>>>>>>> int main(void) { > >>>>>>>> _check_HYPRE_IJMatrixCreate(); > >>>>>>>> return 0; > >>>>>>>> } > >>>>>>>> <<<<<<< > >>>>>>>> > >>>>>>>> Note - it does not include 'HYPRE.h' here - but redefines the > prototype as 'char HYPRE_IJMatrixCreate(); > >>>>>>>> > >>>>>>>> Compiling it manually: > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> [balay at pj01 petsc]$ cat conftest.c > >>>>>>>> char HYPRE_IJMatrixCreate(); > >>>>>>>> static void _check_HYPRE_IJMatrixCreate() { > HYPRE_IJMatrixCreate(); } > >>>>>>>> > >>>>>>>> int main(void) { > >>>>>>>> _check_HYPRE_IJMatrixCreate(); > >>>>>>>> return 0; > >>>>>>>> } > >>>>>>>> [balay at pj01 petsc]$ gcc -c conftest.c > >>>>>>>> [balay at pj01 petsc]$ nm -Ao conftest.o |grep HYPRE_IJMatrixCreate > >>>>>>>> conftest.o:0000000000000000 t _check_HYPRE_IJMatrixCreate > >>>>>>>> conftest.o: U HYPRE_IJMatrixCreate > >>>>>>>> [balay at pj01 petsc]$ nm -Ao arch-linux-c-debug/lib/libHYPRE.so > |grep HYPRE_IJMatrixCreate > >>>>>>>> arch-linux-c-debug/lib/libHYPRE.so:000000000007f2c2 T > HYPRE_IJMatrixCreate > >>>>>>>> [balay at pj01 petsc]$ > >>>>>>>> <<<< > >>>>>>>> > >>>>>>>> Here the "U HYPRE_IJMatrixCreate" in conftest.o matches "T > HYPRE_IJMatrixCreate" in libHYPRE.so - so the "link" test in configure > succeeds! > >>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>> [balay at pj01 petsc]$ gcc -o conftest conftest.o > arch-linux-c-debug/lib/libHYPRE.so > >>>>>>>> [balay at pj01 petsc]$ echo $? > >>>>>>>> 0 > >>>>>>>> <<<<< > >>>>>>>> > >>>>>>>> On windows - [due to name mangling by cdecl/stdcall, (/MT vs /MD) > etc..] - this might not match - resulting in link failures. > >>>>>>>> > >>>>>>>> Satish > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, 19 Jul 2023, Satish Balay via petsc-users wrote: > >>>>>>>> > >>>>>>>>> You could try skipping this test [and assume > --with-hypre-include and --with-hypre-lib options are correct] - and see if > this works. > >>>>>>>>> > >>>>>>>>> diff --git a/config/BuildSystem/config/packages/hypre.py > b/config/BuildSystem/config/packages/hypre.py > >>>>>>>>> index 5bc88322aa2..2d6c7932e17 100644 > >>>>>>>>> --- a/config/BuildSystem/config/packages/hypre.py > >>>>>>>>> +++ b/config/BuildSystem/config/packages/hypre.py > >>>>>>>>> @@ -11,7 +11,7 @@ class Configure(config.package.GNUPackage): > >>>>>>>>> self.requiresversion = 1 > >>>>>>>>> self.gitcommit = 'v'+self.version > >>>>>>>>> self.download = ['git:// > https://github.com/hypre-space/hypre',' > https://github.com/hypre-space/hypre/archive/'+self.gitcommit+'.tar.gz'] > >>>>>>>>> - self.functions = ['HYPRE_IJMatrixCreate'] > >>>>>>>>> + self.functions = [] > >>>>>>>>> self.includes = ['HYPRE.h'] > >>>>>>>>> self.liblist = [['libHYPRE.a']] > >>>>>>>>> self.buildLanguages = ['C','Cxx'] > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Satish > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, 19 Jul 2023, Barry Smith wrote: > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> You don't indicate what type of libraries you built hypre with; > static or shared. My guess is you ended up with shared > >>>>>>>>>> > >>>>>>>>>> I think the answer to your difficulty is hidden in __cdecl > (Satish will know much better than me). When you are looking for symbols in > Windows shared libraries you have to prepend something to the function > prototype to have it successfully found. For example the PETSc include > files have these things __declspec(dllimport) The configure test fails > because it does not provide the needed prototype. Likely you built PTScotch > with static libraries so no problem. > >>>>>>>>>> > >>>>>>>>>> The simplest fix would be to build static hypre libraries. I > think it is a major project to get PETSc configure and macro system to work > properly with external packages that are in Windows shared libraries since > more use of __declspec would be needed. > >>>>>>>>>> > >>>>>>>>>> Barry > >>>>>>>>>> > >>>>>>>>>> The PETSc installation instructions should probably say > something about external packages with Windows shared libraries. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Jul 19, 2023, at 10:52 AM, Daniel Stone < > daniel.stone at opengosim.com > wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hello, > >>>>>>>>>>> > >>>>>>>>>>> I'm working on getting a petsc build running on windows. One > necessary package to include is Hypre. I've been able to build Hypre > seperately using cmake, and confirmed that the library works > >>>>>>>>>>> by setting up a VS project to run some of the example programs. > >>>>>>>>>>> > >>>>>>>>>>> My attempted petsc build is being done through cygwin. I've > been able to (with varying degrees of difficulty), build a fairly plain > petsc, and one that downloads and builds ptscotch (after some modifications > >>>>>>>>>>> to both ptscotch and the config script). I am now attempting > to include Hypre (using the --hypre-iclude and --hypre-lib flags, etc). > Note that the same compilers are being used for both Hypre and for petsc > >>>>>>>>>>> through cygwin - the new intel oneapi compilers (icx and ifx, > after again varying amounts of pain to work around their awkwardness with > the config script). > >>>>>>>>>>> > >>>>>>>>>>> I'm seeing a problem when the config script does some tests on > the included hypre lib. The source code looks like: > >>>>>>>>>>> > >>>>>>>>>>> #include "confdefs.h" > >>>>>>>>>>> #include "conffix.h" > >>>>>>>>>>> /* Override any gcc2 internal prototype to avoid an error. */ > >>>>>>>>>>> > >>>>>>>>>>> #include "HYPRE.h" > >>>>>>>>>>> > >>>>>>>>>>> char HYPRE_IJMatrixCreate(); > >>>>>>>>>>> static void _check_HYPRE_IJMatrixCreate() { > HYPRE_IJMatrixCreate(); } > >>>>>>>>>>> > >>>>>>>>>>> int main() { > >>>>>>>>>>> _check_HYPRE_IJMatrixCreate();; > >>>>>>>>>>> return 0; > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> As I understand this is a fairly standard type of stub program > used by the config script to check that it is able to link to certain > symbols in given libraries. Tests like this have succeeded in my builds that > >>>>>>>>>>> include PTScotch. > >>>>>>>>>>> > >>>>>>>>>>> I keep getting a linker error with the above test, including > if I seperate it out and try to build it seperately: > >>>>>>>>>>> > >>>>>>>>>>> unresolved external symbol "char __cdel > HYPRE_IJMatrixCreate(void)" .... > >>>>>>>>>>> > >>>>>>>>>>> Ok, it looks like a problem with either the library or linker > commands. But here's the interesting thing - If I transplant this code into > VS, with the same project setting that allows it to build the much more > >>>>>>>>>>> nontrivial Hypre example programs, I get the same error: > >>>>>>>>>>> > >>>>>>>>>>> Error LNK2001 unresolved external symbol "char __cdecl > HYPRE_IJMatrixCreate(void)" (?HYPRE_IJMatrixCreate@@YADXZ) hypretry1 > C:\Users\DanielOGS\source\repos\hypretry1\hypretry1\Source.obj 1 > >>>>>>>>>>> > >>>>>>>>>>> So it seems like there might be something about this type of > stub program that is not working with my Hypre library. I don't fully > understand this program - it's able to call the function with no arguments, > but > >>>>>>>>>>> it also needs to be linked against a library containing the > function, apparently by wrapping it in a static void function? Not > something I've seen before. > >>>>>>>>>>> > >>>>>>>>>>> Does anyone have any insight into what might be going wrong - > or really just any explaination of how the stub program works so I can > figure out why it isn't in this case? > >>>>>>>>>>> > >>>>>>>>>>> Many thanks, > >>>>>>>>>>> > >>>>>>>>>>> Daniel > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.stone at opengosim.com Mon Jul 24 05:34:22 2023 From: daniel.stone at opengosim.com (Daniel Stone) Date: Mon, 24 Jul 2023 11:34:22 +0100 Subject: [petsc-users] support for mixed block size matrices/AIM in PETSc? Message-ID: Hello PETSc Users/Developers, A collegue of mine is looking into implementing an adaptive implicit method (AIM) over PETSc in our simulator. This has led to some interesting questions about what can be done with blocked matrices, which I'm not able to answer myself - does anyone have any insight? Apparently it would be ideal if we could find a matrix (and vector) type that supports a kind of mixed block size: "For AIM [...] we will have matrix elements of various shapes: 1x1, 1xN, Nx1 and NxN. [...]. The solution and residual will be a mix of 1 and N variable/cell block" There are ideas for how to implement what we want using the fixed-block-size objects we understand well, but if anything like the above exists it would be very exciting. Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jul 24 07:33:11 2023 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Jul 2023 07:33:11 -0500 (CDT) Subject: [petsc-users] support for mixed block size matrices/AIM in PETSc? In-Reply-To: References: Message-ID: <9d6d8172-2b36-9cb3-fb6a-03be1396fab0@mcs.anl.gov> One way to boost performance [of MatVec etc] in sparse matrices with blocks is by avoiding loading (from memory to cpu registers) of row/col indices for the blocks - when possible. [the performance boost here come by the fact that the memory bandwidth requirements get reduced] So we have BAIJ matrix - where only one row,col value is used for a block of values. And as you've noted - this format requires a fixed block size for the entire matrix. Alternative support in PETSc is within AIJ matrix (enabled by default) - where it scans consecutive rows - to see if they share the same column values (i.e inodes) - and reuses them for these rows. A matrix with varying blocks might benefit from this. To check - for ex: [balay at pj01 tutorials]$ ./ex10 -f0 ~/datafiles/matrices/cfd.2.10 -f1 ~/datafiles/matrices/cfd.2.10 -info |grep -i inode [0] MatSeqAIJCheckInode(): Found 24576 nodes of 122880. Limit used: 5. Using Inode routines [0] MatSeqAIJCheckInode(): Found 24576 nodes of 122880. Limit used: 5. Using Inode routines Satish On Mon, 24 Jul 2023, Daniel Stone wrote: > Hello PETSc Users/Developers, > > A collegue of mine is looking into implementing an adaptive implicit method > (AIM) over > PETSc in our simulator. This has led to some interesting questions about > what can > be done with blocked matrices, which I'm not able to answer myself - does > anyone have > any insight? > > Apparently it would be ideal if we could find a matrix (and vector) type > that supports a kind > of mixed block size: > > "For AIM [...] we will have matrix elements of various shapes: 1x1, 1xN, > Nx1 and NxN. [...]. The solution and residual will be a mix of 1 and N > variable/cell block" > > There are ideas for how to implement what we want using the > fixed-block-size objects we > understand well, but if anything like the above exists it would be very > exciting. > > > Thanks, > > Daniel > From knepley at gmail.com Mon Jul 24 08:29:13 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Jul 2023 09:29:13 -0400 Subject: [petsc-users] support for mixed block size matrices/AIM in PETSc? In-Reply-To: References: Message-ID: On Mon, Jul 24, 2023 at 6:34?AM Daniel Stone wrote: > Hello PETSc Users/Developers, > > A collegue of mine is looking into implementing an adaptive implicit > method (AIM) over > PETSc in our simulator. This has led to some interesting questions about > what can > be done with blocked matrices, which I'm not able to answer myself - does > anyone have > any insight? > > Apparently it would be ideal if we could find a matrix (and vector) type > that supports a kind > of mixed block size: > > "For AIM [...] we will have matrix elements of various shapes: 1x1, 1xN, > Nx1 and NxN. [...]. The solution and residual will be a mix of 1 and N > variable/cell block" > This is not the terminology we would use, since "blocksize" is usually understood as something small that is used for vectorization and indexing. These are large, O(N), and require different implementation. To build matrices out of these parts, you can use 1) MatNest: Very small blocks are not ideal here, so 2) MatLRC: This is for low-rank corrections, which is where small matrices tend to arise Does this make sense? Thanks, Matt > There are ideas for how to implement what we want using the > fixed-block-size objects we > understand well, but if anything like the above exists it would be very > exciting. > > > Thanks, > > Daniel > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From itugrik76 at mail.ru Mon Jul 24 12:11:42 2023 From: itugrik76 at mail.ru (=?UTF-8?B?0JrQvtC90YHRgtCw0L3RgtC40L0=?=) Date: Mon, 24 Jul 2023 20:11:42 +0300 Subject: [petsc-users] (no subject) Message-ID: <1690218702.902583166@f747.i.mail.ru> Good evening. After configuring petsc I had to write this comand on cygwin64. $ make PETSC_DIR=/home/itugr/asd/petsc-3.19.3 PETSC_ARCH=arch-mswin-c-opt all But I have such problem makefile:26: /home/itugr/asd/petsc-3.19.3/lib/petsc/conf/rules.utils: No such file or directory make[1]: *** No rule to make target '/home/itugr/asd/petsc-3.19.3/lib/petsc/conf/rules.utils'. ?Stop. make: *** [GNUmakefile:9: all] Error 2 So, i have this directory, but can't make it. itugr at LAPTOP-UJI8JB1K ~/asd/petsc-3.19.3/lib/petsc/conf $ dir bfort-base.txt ?bfort-mpi.txt ?bfort-petsc.txt ?petscvariables ?rules ?rules.doc ?rules.utils ?test ?uncrustify.cfg ?variables -- ?????????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 24 13:07:21 2023 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 24 Jul 2023 14:07:21 -0400 Subject: [petsc-users] (no subject) In-Reply-To: <1690218702.902583166@f747.i.mail.ru> References: <1690218702.902583166@f747.i.mail.ru> Message-ID: Perhaps you need > make PETSC_DIR=~/asd/petsc-3.19.3 PETSC_ARCH=arch-mswin-c-opt all > On Jul 24, 2023, at 1:11 PM, ?????????? via petsc-users wrote: > > Good evening. After configuring petsc I had to write this comand on cygwin64. > $ make PETSC_DIR=/home/itugr/asd/petsc-3.19.3 PETSC_ARCH=arch-mswin-c-opt all > But I have such problem > makefile:26: /home/itugr/asd/petsc-3.19.3/lib/petsc/conf/rules.utils: No such file or directory > make[1]: *** No rule to make target '/home/itugr/asd/petsc-3.19.3/lib/petsc/conf/rules.utils'. Stop. > make: *** [GNUmakefile:9: all] Error 2 > So, i have this directory, but can't make it. > itugr at LAPTOP-UJI8JB1K ~/asd/petsc-3.19.3/lib/petsc/conf > $ dir > bfort-base.txt bfort-mpi.txt bfort-petsc.txt petscvariables rules rules.doc rules.utils test uncrustify.cfg variables > -- > ?????????? From mfadams at lbl.gov Mon Jul 24 13:44:39 2023 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 24 Jul 2023 14:44:39 -0400 Subject: [petsc-users] support for mixed block size matrices/AIM in PETSc? In-Reply-To: References: Message-ID: I interpreted Danial's question as referring to small block sizes for multi-dof per point/vertex/cell in the mesh, like Satish, but please clarify. With that assumption, as you know PETSc does not support explicit variable block sizes, like ML does for instance, but the conventional wisdom has been that inodes do a pretty good job of exploiting block structure for performance as Satish mentioned. It would be great to get some data from modern processors to quantify this. If your solver is in fact memory bandwidth bound, as it should be, then if you test a problem with constant block size >= 3 you should see the solver takes almost 50% longer if you turn inodes off. This test would give some sense of how well inodes are working for you. Mark On Mon, Jul 24, 2023 at 9:43?AM Matthew Knepley wrote: > On Mon, Jul 24, 2023 at 6:34?AM Daniel Stone > wrote: > >> Hello PETSc Users/Developers, >> >> A collegue of mine is looking into implementing an adaptive implicit >> method (AIM) over >> PETSc in our simulator. This has led to some interesting questions about >> what can >> be done with blocked matrices, which I'm not able to answer myself - does >> anyone have >> any insight? >> >> Apparently it would be ideal if we could find a matrix (and vector) type >> that supports a kind >> of mixed block size: >> >> "For AIM [...] we will have matrix elements of various shapes: 1x1, 1xN, >> Nx1 and NxN. [...]. The solution and residual will be a mix of 1 and N >> variable/cell block" >> > > This is not the terminology we would use, since "blocksize" is usually > understood as something small > that is used for vectorization and indexing. These are large, O(N), and > require different implementation. > To build matrices out of these parts, you can use > > 1) MatNest: Very small blocks are not ideal here, so > > 2) MatLRC: This is for low-rank corrections, which is where small > matrices tend to arise > > Does this make sense? > > Thanks, > > Matt > > >> There are ideas for how to implement what we want using the >> fixed-block-size objects we >> understand well, but if anything like the above exists it would be very >> exciting. >> >> >> Thanks, >> >> Daniel >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bui at calcreek.com Mon Jul 24 19:16:09 2023 From: bui at calcreek.com (Thuc Bui) Date: Mon, 24 Jul 2023 17:16:09 -0700 Subject: [petsc-users] 3D Poisson solver failed in KSPSolve when number of cores is larger than one Message-ID: <00dd01d9be8d$36878ea0$a396abe0$@calcreek.com> Dear PETSc Users/Developers. I have been successfully using PETsc on Windows without MPI for a while now. I have now attempted to implement PETSc with MPI on Windows 10. I have built a release version of PETSc 3.18.6 with MS MPI 10.1.2, Intel MKL 3.279 (2020) and Visual Studio 2019 as a static library. I am testing a finite difference 3D Poisson solver (cylindrical coordinates) with this MPI PETSc. On a Windows 10 laptop with 4 cores (8 logical processors), the solver works fine with one core. However, it gives errors with 2 cores. The errors are listed below followed by a complete code. The errors are generated in KSPSolve. I perhaps miss some settings to cause this error "No method muldiagonalblock for Mat of type seqaij". I don't know why for a 2-core MPI program that a seqaij matrix is used in KSPSolve. I would be very grateful if someone can provide me with some direction to fix these issues. Many thanks for your help, Thuc Bui Senior R&D Engineer Calabazas Creek Research, Inc (650) 948-5361 ///////////////errors///////////////////////// rank=0 iStart=0 iEnd=2375 rank=1 iStart=2375 iEnd=4750 [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: [0]PETSC ERROR: No support for this operation for this object type No support for this operation for this object type [1]PETSC ERROR: [0]PETSC ERROR: No method multdiagonalblock for Mat of type seqaij No method multdiagonalblock for Mat of type seqaij [1]PETSC ERROR: [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. See https://petsc.org/release/faq/ for trouble shooting. [1]PETSC ERROR: [0]PETSC ERROR: Petsc Release Version 3.18.6, Mar 30, 2023 Petsc Release Version 3.18.6, Mar 30, 2023 [1]PETSC ERROR: [0]PETSC ERROR: Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 15:16:55 2023 [1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe cl" --with-debugging=0 --with-openmp --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MS MPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MS MPI/latest/include/x64 --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMP I/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpac kages/MSMPI/latest/Lib/x64/msmpifec.lib]" --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackag es/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Document s/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Us ers/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_t hread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/la test/x64/lib/libiomp5md.lib]" --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/ latest/bin/x64/mpiexec --with-shared-libraries=0 Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 15:16:55 2023 [1]PETSC ERROR: [0]PETSC ERROR: #1 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe cl" --with-debugging=0 --with-openmp --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MS MPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MS MPI/latest/include/x64 --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMP I/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpac kages/MSMPI/latest/Lib/x64/msmpifec.lib]" --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackag es/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Document s/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Us ers/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_t hread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/la test/x64/lib/libiomp5md.lib]" --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/ latest/bin/x64/mpiexec --with-shared-libraries=0 [1]PETSC ERROR: [0]PETSC ERROR: #2 MatMultDiagonalBlock_MPIAIJ() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 #1 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 [1]PETSC ERROR: [0]PETSC ERROR: #3 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 #2 MatMultDiagonalBlock_MPIAIJ() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 [1]PETSC ERROR: [0]PETSC ERROR: #4 PCApply_Eisenstat() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 #3 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 [1]PETSC ERROR: [0]PETSC ERROR: #5 PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 #4 PCApply_Eisenstat() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 [1]PETSC ERROR: [0]PETSC ERROR: #6 KSP_PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 #5 PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 [1]PETSC ERROR: [0]PETSC ERROR: #7 KSPInitialResidual() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 #6 KSP_PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 [1]PETSC ERROR: [0]PETSC ERROR: #8 KSPSolve_GMRES() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 #7 KSPInitialResidual() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 [1]PETSC ERROR: [0]PETSC ERROR: #9 KSPSolve_Private() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 #8 KSPSolve_GMRES() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 [1]PETSC ERROR: [0]PETSC ERROR: #10 KSPSolve() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 #9 KSPSolve_Private() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 [1]PETSC ERROR: [0]PETSC ERROR: #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 #10 KSPSolve() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 [1]PETSC ERROR: [0]PETSC ERROR: No PETSc Option Table entries #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- //complete codes/////////////////////////////////////////////// #include "petscsys.h" #include "petscksp.h" #ifndef PETSC_SUCCESS #define PETSC_SUCCESS 0 #endif //rectangular matrix mxn int** allocateIMatrixR(int m, int n) { int** v = (int**)malloc(m * sizeof(int*)); for (int i = 0; i < m; ++i) { v[i] = (int*)malloc(n * sizeof(int)); } for (int i = 0; i < m; ++i) { for (int j = 0; j < n; ++n) { v[i][j] = -1; } } return v; } int mapIndex(int k, int i, int j, int Nz, int Nr) { //k for z, i for r, j for theta //base 0 //3D table: theta r z // 0 0 0 // 0 0 1 // 0 0 2 // ... // 0 1 0 // 0 1 1 // 0 1 2 // ... int n = k + i * Nz + j * Nz * Nr; return n; } PetscErrorCode MPIAssembler(Mat* A, Vec* b, int Kz, int Ir, int Jt) { PetscFunctionBeginUser; int M; PetscCall(VecGetSize(*b, &M)); //contruct an array of Mx3, row index is the equation number, each row is (i,j,k), //i is r-index, j theta-index, k z-index const int dim = 3; int** Nijk = allocateIMatrixR(M, dim); for (int k = 0; k < Kz; ++k) { int k1 = k + 1; for (int i = 0; i < Ir; ++i) { for (int j = 0; j < Jt; ++j) { //for each equation row m int m = mapIndex(k, i, j, Kz, Ir); Nijk[m][0] = i; Nijk[m][1] = j; Nijk[m][2] = k; } } } //////////////////////////////////////////////////////////////////////////// ///// //Cylinder dimensions double R = 11.0; //cm, cylinder radius double L = 11.0; //cm, cylinder length in z double Dr = R / (Ir + 1); double Dz = L / (Kz + 1); double Dr2 = Dr * Dr; double Dr_2 = 0.5 * Dr; double Dr_22 = Dr_2 * Dr_2; double Dz_2 = 1.0 / (Dz * Dz); double DT = 2.0 * M_PI / Jt; double DT2 = DT * DT; double Vr = 0.0; //Dirichlet on R double chargeDens = -4.0; int rank; PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank)); //get the range of matrix rows owned by this processor int startingNumber, endingNumber; PetscCall(MatGetOwnershipRange(*A, &startingNumber, &endingNumber)); fprintf(stderr, "rank=%i iStart=%i iEnd=%i\n", rank, startingNumber, endingNumber); int Ir_1 = Ir - 1; int Jt_1 = Jt - 1; for (int m = startingNumber; m < endingNumber; ++m) { int i = Nijk[m][0]; int j = Nijk[m][1]; int k = Nijk[m][2]; double ri = Dr * (i + 1); double ri_2 = ri - Dr_2; double ri2 = ri + Dr_2; double sqri = ri * ri; double Ai = (ri_2 / ri) / Dr2; double Bi = (ri2 / ri) / Dr2; double Ci = 1.0 / (sqri * DT2); double Ei = (ri2 + ri_2) / (ri * Dr2) + 2.0 / (sqri * DT2) + 2.0 * Dz_2; //contribution of charge density to RHS PetscCall(VecSetValue(*b, m, -chargeDens, ADD_VALUES)); //contribution to RHS by Drichlet on R if (i == Ir_1) { PetscCall(VecSetValue(*b, i, -Bi * Vr, ADD_VALUES)); } if (!i) { //from the axis //contributing to RHS PetscCall(VecSetValue(*b, m, -chargeDens * Ai * Dr_22, ADD_VALUES)); //contributing to LHS double xTheta = (Ai / Jt); for (int jt = 0; jt < Jt; ++jt) { int nj = mapIndex(k, i, jt, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &nj, &xTheta, ADD_VALUES)); } } else { //0 < i < Ir: contribution of Ai to LHS int i_jk = mapIndex(k, i - 1, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &i_jk, &Ai, ADD_VALUES)); } if (i != Ir_1) { //0 <= i < It-1: contribution of Bi to LHS int ki1j = mapIndex(k, i + 1, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &ki1j, &Bi, ADD_VALUES)); } //apply periodic BC in theta//////////////////////////// int kij_; if (!j) { kij_ = mapIndex(k, i, Jt_1, Kz, Ir); } else { kij_ = mapIndex(k, i, j - 1, Kz, Ir); } //1st contribution of Ci to LHS PetscCall(MatSetValues(*A, 1, &m, 1, &kij_, &Ci, ADD_VALUES)); int kij1; if (j == Jt_1) { kij1 = mapIndex(k, i, 0, Kz, Ir); } else { kij1 = mapIndex(k, i, j + 1, Kz, Ir); } //2nd contribution of Ci to LHS PetscCall(MatSetValues(*A, 1, &m, 1, &kij1, &Ci, ADD_VALUES)); //////////////////////////////////////////////// //apply z Neumann BC /////////////////////////////////// int k_ij; if (!k) { //z=0 k_ij = mapIndex(k, i, j, Kz, Ir); } else { //0 < z < L k_ij = mapIndex(k - 1, i, j, Kz, Ir); } PetscCall(MatSetValues(*A, 1, &m, 1, &k_ij, &Dz_2, ADD_VALUES)); ///////////////////////////////////////////////////////// //contribution of Ei to LHS int kij = mapIndex(k, i, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &kij, &Ei, ADD_VALUES)); } if (Nijk) { for (int i = 0; i < M; ++i) { free(Nijk[i]); } free(Nijk); } PetscCall(VecAssemblyBegin(*b)); PetscCall(VecAssemblyEnd(*b)); PetscCall(MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY)); PetscFunctionReturn(PETSC_SUCCESS); } int main(int argc, char** argv) { PetscFunctionBeginUser; const char help[] = "Poisson3D solver using PETSc."; PetscCall(PetscInitialize(&argc, &argv, PETSC_NULL, help)); Mat mat_; Vec B_; Vec X_; int Nz = 26; //number of z-segments int Nr = 20; //number of r-segments int Nt = 10; //number of theta-segments PetscCall(PetscOptionsGetInt(NULL, NULL, "-z", &Nz, NULL)); PetscCall(PetscOptionsGetInt(NULL, NULL, "-r", &Nr, NULL)); PetscCall(PetscOptionsGetInt(NULL, NULL, "-t", &Nt, NULL)); //Neumann at z=0 and z=L //Dirichlet at R, the cylinder radius //On axis, compute field //number of unknowns of cylinder interior nodes minus axis int Kz = Nz - 1; int Ir = Nr - 1; int M = Kz * Ir * Nt; int bw = 15; //for 7-point stencil PetscCall(MatCreate(PETSC_COMM_WORLD, &mat_)); PetscCall(MatSetSizes(mat_, PETSC_DECIDE, PETSC_DECIDE, M, M)); PetscCall(MatSetFromOptions(mat_)); PetscCall(MatMPIAIJSetPreallocation(mat_, bw, 0, bw, 0)); PetscCall(MatSeqAIJSetPreallocation(mat_, bw, 0)); //source PetscCall(VecCreate(PETSC_COMM_WORLD, &B_)); PetscCall(VecSetSizes(B_, PETSC_DECIDE, M)); PetscCall(VecSetFromOptions(B_)); //sol PetscCall(VecDuplicate(B_, &X_)); MPIAssembler(&mat_, &B_, Kz, Ir, Nt); PCType pcType = PCEISENSTAT; KSPType solverType = KSPGMRES; int maxIters = 100; double relTol = 1.0e-6; KSP ksp; PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); PetscCall(KSPSetOperators(ksp, mat_, mat_)); PetscCall(KSPSetType(ksp, solverType)); //preconditioner PC pc; PetscCall(KSPGetPC(ksp, &pc)); PetscCall(PCSetType(pc, pcType)); PetscCall(KSPSetTolerances(ksp, relTol, PETSC_DEFAULT, PETSC_DEFAULT, maxIters)); PetscCall(KSPSetFromOptions(ksp)); //solving PetscCall(KSPSolve(ksp, B_, X_)); double xMin, xMax; int mnIndx, mxIndx; PetscCall(VecMin(X_, &mnIndx, &xMin)); PetscCall(VecMax(X_, &mxIndx, &xMax)); fprintf(stderr, "mnIndx=%i mxIndx=%i xMin=%g xMax=%g\n", mnIndx, mxIndx, xMin, xMax); PetscCall(PetscFinalize()); return 0; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 24 20:49:37 2023 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Jul 2023 21:49:37 -0400 Subject: [petsc-users] 3D Poisson solver failed in KSPSolve when number of cores is larger than one In-Reply-To: <00dd01d9be8d$36878ea0$a396abe0$@calcreek.com> References: <00dd01d9be8d$36878ea0$a396abe0$@calcreek.com> Message-ID: On Mon, Jul 24, 2023 at 8:16?PM Thuc Bui wrote: > Dear PETSc Users/Developers. > > > > I have been successfully using PETsc on Windows without MPI for a while > now. I have now attempted to implement PETSc with MPI on Windows 10. I have > built a release version of PETSc 3.18.6 with MS MPI 10.1.2, Intel MKL 3.279 > (2020) and Visual Studio 2019 as a static library. I am testing a finite > difference 3D Poisson solver (cylindrical coordinates) with this MPI PETSc. > > > > On a Windows 10 laptop with 4 cores (8 logical processors), the solver > works fine with one core. However, it gives errors with 2 cores. The errors > are listed below followed by a complete code. The errors are generated in > KSPSolve. I perhaps miss some settings to cause this error ?No method > muldiagonalblock for Mat of type seqaij?. I don?t know why for a 2-core MPI > program that a seqaij matrix is used in KSPSolve. > > > > I would be very grateful if someone can provide me with some direction to > fix these issues. > That is a bug with PCEISENSTAT. It is not frequently used. Since your matrix does not use inodes, a default implementation of MatMultDIagonalBlock is not found. We just need to add this for MATSEQAIJ. It should work with a different PC. We will fix this as soon as we can. Thanks, Matt > Many thanks for your help, > > Thuc Bui > > Senior R&D Engineer > > Calabazas Creek Research, Inc > > (650) 948-5361 > > > > > > ///////////////errors///////////////////////// > > rank=0 iStart=0 iEnd=2375 > > rank=1 iStart=2375 iEnd=4750 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > --------------------- Error Message > -------------------------------------------------------------- > > --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > No support for this operation for this object type > > No support for this operation for this object type > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > No method multdiagonalblock for Mat of type seqaij > > No method multdiagonalblock for Mat of type seqaij > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > See https://petsc.org/release/faq/ for trouble shooting. > > See https://petsc.org/release/faq/ for trouble shooting. > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > Petsc Release Version 3.18.6, Mar 30, 2023 > > Petsc Release Version 3.18.6, Mar 30, 2023 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 > 15:16:55 2023 > > [1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0 > --with-cxx="win32fe cl" --with-debugging=0 --with-openmp > --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64 > --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]" > --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]" > --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec > --with-shared-libraries=0 > > Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 > 15:16:55 2023 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #1 MatMultDiagonalBlock() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 > > Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe > cl" --with-debugging=0 --with-openmp > --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64 > --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]" > --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]" > --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec > --with-shared-libraries=0 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #2 MatMultDiagonalBlock_MPIAIJ() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 > > #1 MatMultDiagonalBlock() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #3 MatMultDiagonalBlock() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 > > #2 MatMultDiagonalBlock_MPIAIJ() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #4 PCApply_Eisenstat() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 > > #3 MatMultDiagonalBlock() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #5 PCApply() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 > > #4 PCApply_Eisenstat() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #6 KSP_PCApply() at > C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 > > #5 PCApply() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #7 KSPInitialResidual() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 > > #6 KSP_PCApply() at > C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #8 KSPSolve_GMRES() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 > > #7 KSPInitialResidual() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #9 KSPSolve_Private() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 > > #8 KSPSolve_GMRES() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #10 KSPSolve() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 > > #9 KSPSolve_Private() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 > > #10 KSPSolve() at > C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > No PETSc Option Table entries > > #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 > > [1]PETSC ERROR: > > [0]PETSC ERROR: > > ----------------End of Error Message -------send entire error message to > petsc-maint at mcs.anl.gov---------- > > No PETSc Option Table entries > > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > > > //complete codes/////////////////////////////////////////////// > > #include "petscsys.h" > > #include "petscksp.h" > > #ifndef PETSC_SUCCESS > > #define PETSC_SUCCESS 0 > > #endif > > //rectangular matrix mxn > > int** allocateIMatrixR(int m, int n) > > { > > int** v = (int**)malloc(m * sizeof(int*)); > > for (int i = 0; i < m; ++i) { > > v[i] = (int*)malloc(n * sizeof(int)); > > } > > for (int i = 0; i < m; ++i) { > > for (int j = 0; j < n; ++n) { > > v[i][j] = -1; > > } > > } > > return v; > > } > > int mapIndex(int k, int i, int j, int Nz, int Nr) > > { > > //k for z, i for r, j for theta > > //base 0 > > //3D table: theta r z > > // 0 0 0 > > // 0 0 1 > > // 0 0 2 > > // ... > > // 0 1 0 > > // 0 1 1 > > // 0 1 2 > > // ... > > int n = k + i * Nz + j * Nz * Nr; > > return n; > > } > > PetscErrorCode MPIAssembler(Mat* A, Vec* b, int Kz, int Ir, int Jt) > > { > > PetscFunctionBeginUser; > > int M; > > PetscCall(VecGetSize(*b, &M)); > > > //contruct an array of Mx3, row index is the equation number, each row is (i,j,k), > > //i is r-index, j theta-index, k z-index > > const int dim = 3; > > int** Nijk = allocateIMatrixR(M, dim); > > for (int k = 0; k < Kz; ++k) { > > int k1 = k + 1; > > for (int i = 0; i < Ir; ++i) { > > for (int j = 0; j < Jt; ++j) { > > //for each equation row m > > int m = mapIndex(k, i, j, Kz, Ir); > > Nijk[m][0] = i; > > Nijk[m][1] = j; > > Nijk[m][2] = k; > > } > > } > > } > > > ///////////////////////////////////////////////////////////////////////////////// > > //Cylinder dimensions > > double R = 11.0; //cm, cylinder radius > > double L = 11.0; //cm, cylinder length in z > > double Dr = R / (Ir + 1); > > double Dz = L / (Kz + 1); > > double Dr2 = Dr * Dr; > > double Dr_2 = 0.5 * Dr; > > double Dr_22 = Dr_2 * Dr_2; > > double Dz_2 = 1.0 / (Dz * Dz); > > double DT = 2.0 * M_PI / Jt; > > double DT2 = DT * DT; > > double Vr = 0.0; //Dirichlet on R > > double chargeDens = -4.0; > > int rank; > > PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank)); > > //get the range of matrix rows owned by this processor > > int startingNumber, endingNumber; > > > PetscCall(MatGetOwnershipRange(*A, &startingNumber, &endingNumber)); > > fprintf(stderr, "rank=%i iStart=%i iEnd=%i\n" > , rank, startingNumber, endingNumber); > > int Ir_1 = Ir - 1; > > int Jt_1 = Jt - 1; > > for (int m = startingNumber; m < endingNumber; ++m) { > > int i = Nijk[m][0]; > > int j = Nijk[m][1]; > > int k = Nijk[m][2]; > > double ri = Dr * (i + 1); > > double ri_2 = ri - Dr_2; > > double ri2 = ri + Dr_2; > > double sqri = ri * ri; > > double Ai = (ri_2 / ri) / Dr2; > > double Bi = (ri2 / ri) / Dr2; > > double Ci = 1.0 / (sqri * DT2); > > double > Ei = (ri2 + ri_2) / (ri * Dr2) + 2.0 / (sqri * DT2) + 2.0 * Dz_2; > > //contribution of charge density to RHS > > PetscCall(VecSetValue(*b, m, -chargeDens, ADD_VALUES)); > > //contribution to RHS by Drichlet on R > > if (i == Ir_1) { > > > PetscCall(VecSetValue(*b, i, -Bi * Vr, ADD_VALUES)); > > } > > > > if (!i) { //from the axis > > //contributing to RHS > > > PetscCall(VecSetValue(*b, m, -chargeDens * Ai * Dr_22, ADD_VALUES)); > > //contributing to LHS > > double xTheta = (Ai / Jt); > > for (int jt = 0; jt < Jt; ++jt) { > > int nj = mapIndex(k, i, jt, Kz, Ir); > > > PetscCall(MatSetValues(*A, 1, &m, 1, &nj, &xTheta, ADD_VALUES)); > > } > > } > > else { > > //0 < i < Ir: contribution of Ai to LHS > > int i_jk = mapIndex(k, i - 1, j, Kz, Ir); > > > PetscCall(MatSetValues(*A, 1, &m, 1, &i_jk, &Ai, ADD_VALUES)); > > } > > if (i != Ir_1) { > > //0 <= i < It-1: contribution of Bi to LHS > > int ki1j = mapIndex(k, i + 1, j, Kz, Ir); > > > PetscCall(MatSetValues(*A, 1, &m, 1, &ki1j, &Bi, ADD_VALUES)); > > } > > //apply periodic BC in theta//////////////////////////// > > int kij_; > > if (!j) { > > kij_ = mapIndex(k, i, Jt_1, Kz, Ir); > > } > > else { > > kij_ = mapIndex(k, i, j - 1, Kz, Ir); > > } > > //1st contribution of Ci to LHS > > > PetscCall(MatSetValues(*A, 1, &m, 1, &kij_, &Ci, ADD_VALUES)); > > int kij1; > > if (j == Jt_1) { > > kij1 = mapIndex(k, i, 0, Kz, Ir); > > } > > else { > > kij1 = mapIndex(k, i, j + 1, Kz, Ir); > > } > > //2nd contribution of Ci to LHS > > > PetscCall(MatSetValues(*A, 1, &m, 1, &kij1, &Ci, ADD_VALUES)); > > //////////////////////////////////////////////// > > //apply z Neumann BC /////////////////////////////////// > > int k_ij; > > if (!k) { //z=0 > > k_ij = mapIndex(k, i, j, Kz, Ir); > > } > > else { //0 < z < L > > k_ij = mapIndex(k - 1, i, j, Kz, Ir); > > } > > > PetscCall(MatSetValues(*A, 1, &m, 1, &k_ij, &Dz_2, ADD_VALUES)); > > ///////////////////////////////////////////////////////// > > //contribution of Ei to LHS > > int kij = mapIndex(k, i, j, Kz, Ir); > > > PetscCall(MatSetValues(*A, 1, &m, 1, &kij, &Ei, ADD_VALUES)); > > } > > if (Nijk) { > > for (int i = 0; i < M; ++i) { > > free(Nijk[i]); > > } > > free(Nijk); > > } > > PetscCall(VecAssemblyBegin(*b)); > > PetscCall(VecAssemblyEnd(*b)); > > PetscCall(MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY)); > > PetscCall(MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY)); > > PetscFunctionReturn(PETSC_SUCCESS); > > } > > int main(int argc, char** argv) > > { > > PetscFunctionBeginUser; > > const char help[] = "Poisson3D solver using PETSc."; > > PetscCall(PetscInitialize(&argc, &argv, PETSC_NULL, help)); > > Mat mat_; > > Vec B_; > > Vec X_; > > int Nz = 26; //number of z-segments > > int Nr = 20; //number of r-segments > > int Nt = 10; //number of theta-segments > > PetscCall(PetscOptionsGetInt(NULL, NULL, "-z", &Nz, NULL)); > > PetscCall(PetscOptionsGetInt(NULL, NULL, "-r", &Nr, NULL)); > > PetscCall(PetscOptionsGetInt(NULL, NULL, "-t", &Nt, NULL)); > > //Neumann at z=0 and z=L > > //Dirichlet at R, the cylinder radius > > //On axis, compute field > > //number of unknowns of cylinder interior nodes minus axis > > int Kz = Nz - 1; > > int Ir = Nr - 1; > > int M = Kz * Ir * Nt; > > int bw = 15; //for 7-point stencil > > PetscCall(MatCreate(PETSC_COMM_WORLD, &mat_)); > > PetscCall(MatSetSizes(mat_, PETSC_DECIDE, PETSC_DECIDE, M, M)); > > PetscCall(MatSetFromOptions(mat_)); > > PetscCall(MatMPIAIJSetPreallocation(mat_, bw, 0, bw, 0)); > > PetscCall(MatSeqAIJSetPreallocation(mat_, bw, 0)); > > //source > > PetscCall(VecCreate(PETSC_COMM_WORLD, &B_)); > > PetscCall(VecSetSizes(B_, PETSC_DECIDE, M)); > > PetscCall(VecSetFromOptions(B_)); > > //sol > > PetscCall(VecDuplicate(B_, &X_)); > > MPIAssembler(&mat_, &B_, Kz, Ir, Nt); > > PCType pcType = PCEISENSTAT; > > KSPType solverType = KSPGMRES; > > int maxIters = 100; > > double relTol = 1.0e-6; > > KSP ksp; > > PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); > > PetscCall(KSPSetOperators(ksp, mat_, mat_)); > > PetscCall(KSPSetType(ksp, solverType)); > > //preconditioner > > PC pc; > > PetscCall(KSPGetPC(ksp, &pc)); > > PetscCall(PCSetType(pc, pcType)); > > > PetscCall(KSPSetTolerances(ksp, relTol, PETSC_DEFAULT, PETSC_DEFAULT, maxIters)); > > PetscCall(KSPSetFromOptions(ksp)); > > //solving > > PetscCall(KSPSolve(ksp, B_, X_)); > > double xMin, xMax; > > int mnIndx, mxIndx; > > PetscCall(VecMin(X_, &mnIndx, &xMin)); > > PetscCall(VecMax(X_, &mxIndx, &xMax)); > > fprintf(stderr, "mnIndx=%i mxIndx=%i xMin=%g xMax=%g\n" > , mnIndx, mxIndx, xMin, xMax); > > > > PetscCall(PetscFinalize()); > > return 0; > > } > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bui at calcreek.com Mon Jul 24 22:37:09 2023 From: bui at calcreek.com (Thuc Bui) Date: Mon, 24 Jul 2023 20:37:09 -0700 Subject: [petsc-users] 3D Poisson solver failed in KSPSolve when number of cores is larger than one In-Reply-To: References: <00dd01d9be8d$36878ea0$a396abe0$@calcreek.com> Message-ID: <010001d9bea9$4b623ca0$e226b5e0$@calcreek.com> Thank you so much Matt, for getting back to me so quickly. Yes, using another PC fixes the issues. Many thanks again for your help, Thuc From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Monday, July 24, 2023 6:50 PM To: Thuc Bui Cc: petsc-users Subject: Re: [petsc-users] 3D Poisson solver failed in KSPSolve when number of cores is larger than one On Mon, Jul 24, 2023 at 8:16?PM Thuc Bui wrote: Dear PETSc Users/Developers. I have been successfully using PETsc on Windows without MPI for a while now. I have now attempted to implement PETSc with MPI on Windows 10. I have built a release version of PETSc 3.18.6 with MS MPI 10.1.2, Intel MKL 3.279 (2020) and Visual Studio 2019 as a static library. I am testing a finite difference 3D Poisson solver (cylindrical coordinates) with this MPI PETSc. On a Windows 10 laptop with 4 cores (8 logical processors), the solver works fine with one core. However, it gives errors with 2 cores. The errors are listed below followed by a complete code. The errors are generated in KSPSolve. I perhaps miss some settings to cause this error ?No method muldiagonalblock for Mat of type seqaij?. I don?t know why for a 2-core MPI program that a seqaij matrix is used in KSPSolve. I would be very grateful if someone can provide me with some direction to fix these issues. That is a bug with PCEISENSTAT. It is not frequently used. Since your matrix does not use inodes, a default implementation of MatMultDIagonalBlock is not found. We just need to add this for MATSEQAIJ. It should work with a different PC. We will fix this as soon as we can. Thanks, Matt Many thanks for your help, Thuc Bui Senior R&D Engineer Calabazas Creek Research, Inc (650) 948-5361 ///////////////errors///////////////////////// rank=0 iStart=0 iEnd=2375 rank=1 iStart=2375 iEnd=4750 [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: [0]PETSC ERROR: No support for this operation for this object type No support for this operation for this object type [1]PETSC ERROR: [0]PETSC ERROR: No method multdiagonalblock for Mat of type seqaij No method multdiagonalblock for Mat of type seqaij [1]PETSC ERROR: [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. See https://petsc.org/release/faq/ for trouble shooting. [1]PETSC ERROR: [0]PETSC ERROR: Petsc Release Version 3.18.6, Mar 30, 2023 Petsc Release Version 3.18.6, Mar 30, 2023 [1]PETSC ERROR: [0]PETSC ERROR: Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 15:16:55 2023 [1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe cl" --with-debugging=0 --with-openmp --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64 --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]" --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]" --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec --with-shared-libraries=0 Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24 15:16:55 2023 [1]PETSC ERROR: [0]PETSC ERROR: #1 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe cl" --with-debugging=0 --with-openmp --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64 --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]" --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]" --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec --with-shared-libraries=0 [1]PETSC ERROR: [0]PETSC ERROR: #2 MatMultDiagonalBlock_MPIAIJ() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 #1 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 [1]PETSC ERROR: [0]PETSC ERROR: #3 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 #2 MatMultDiagonalBlock_MPIAIJ() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978 [1]PETSC ERROR: [0]PETSC ERROR: #4 PCApply_Eisenstat() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 #3 MatMultDiagonalBlock() at C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538 [1]PETSC ERROR: [0]PETSC ERROR: #5 PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 #4 PCApply_Eisenstat() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\impls\eisens\eisen.c:51 [1]PETSC ERROR: [0]PETSC ERROR: #6 KSP_PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 #5 PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\pc\INTERF~1\precon.c:441 [1]PETSC ERROR: [0]PETSC ERROR: #7 KSPInitialResidual() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 #6 KSP_PCApply() at C:\Users\bui\DOCUME~1\Petsc\latest\include\petsc/private/kspimpl.h:380 [1]PETSC ERROR: [0]PETSC ERROR: #8 KSPSolve_GMRES() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 #7 KSPInitialResidual() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itres.c:64 [1]PETSC ERROR: [0]PETSC ERROR: #9 KSPSolve_Private() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 #8 KSPSolve_GMRES() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\impls\gmres\gmres.c:227 [1]PETSC ERROR: [0]PETSC ERROR: #10 KSPSolve() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 #9 KSPSolve_Private() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:899 [1]PETSC ERROR: [0]PETSC ERROR: #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 #10 KSPSolve() at C:\Users\bui\DOCUME~1\Petsc\latest\src\ksp\ksp\INTERF~1\itfunc.c:1071 [1]PETSC ERROR: [0]PETSC ERROR: No PETSc Option Table entries #11 main() at C:\Users\bui\Documents\Nemesis\Poisson3D\Main.cpp:253 [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- //complete codes/////////////////////////////////////////////// #include "petscsys.h" #include "petscksp.h" #ifndef PETSC_SUCCESS #define PETSC_SUCCESS 0 #endif //rectangular matrix mxn int** allocateIMatrixR(int m, int n) { int** v = (int**)malloc(m * sizeof(int*)); for (int i = 0; i < m; ++i) { v[i] = (int*)malloc(n * sizeof(int)); } for (int i = 0; i < m; ++i) { for (int j = 0; j < n; ++n) { v[i][j] = -1; } } return v; } int mapIndex(int k, int i, int j, int Nz, int Nr) { //k for z, i for r, j for theta //base 0 //3D table: theta r z // 0 0 0 // 0 0 1 // 0 0 2 // ... // 0 1 0 // 0 1 1 // 0 1 2 // ... int n = k + i * Nz + j * Nz * Nr; return n; } PetscErrorCode MPIAssembler(Mat* A, Vec* b, int Kz, int Ir, int Jt) { PetscFunctionBeginUser; int M; PetscCall(VecGetSize(*b, &M)); //contruct an array of Mx3, row index is the equation number, each row is (i,j,k), //i is r-index, j theta-index, k z-index const int dim = 3; int** Nijk = allocateIMatrixR(M, dim); for (int k = 0; k < Kz; ++k) { int k1 = k + 1; for (int i = 0; i < Ir; ++i) { for (int j = 0; j < Jt; ++j) { //for each equation row m int m = mapIndex(k, i, j, Kz, Ir); Nijk[m][0] = i; Nijk[m][1] = j; Nijk[m][2] = k; } } } ///////////////////////////////////////////////////////////////////////////////// //Cylinder dimensions double R = 11.0; //cm, cylinder radius double L = 11.0; //cm, cylinder length in z double Dr = R / (Ir + 1); double Dz = L / (Kz + 1); double Dr2 = Dr * Dr; double Dr_2 = 0.5 * Dr; double Dr_22 = Dr_2 * Dr_2; double Dz_2 = 1.0 / (Dz * Dz); double DT = 2.0 * M_PI / Jt; double DT2 = DT * DT; double Vr = 0.0; //Dirichlet on R double chargeDens = -4.0; int rank; PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank)); //get the range of matrix rows owned by this processor int startingNumber, endingNumber; PetscCall(MatGetOwnershipRange(*A, &startingNumber, &endingNumber)); fprintf(stderr, "rank=%i iStart=%i iEnd=%i\n", rank, startingNumber, endingNumber); int Ir_1 = Ir - 1; int Jt_1 = Jt - 1; for (int m = startingNumber; m < endingNumber; ++m) { int i = Nijk[m][0]; int j = Nijk[m][1]; int k = Nijk[m][2]; double ri = Dr * (i + 1); double ri_2 = ri - Dr_2; double ri2 = ri + Dr_2; double sqri = ri * ri; double Ai = (ri_2 / ri) / Dr2; double Bi = (ri2 / ri) / Dr2; double Ci = 1.0 / (sqri * DT2); double Ei = (ri2 + ri_2) / (ri * Dr2) + 2.0 / (sqri * DT2) + 2.0 * Dz_2; //contribution of charge density to RHS PetscCall(VecSetValue(*b, m, -chargeDens, ADD_VALUES)); //contribution to RHS by Drichlet on R if (i == Ir_1) { PetscCall(VecSetValue(*b, i, -Bi * Vr, ADD_VALUES)); } if (!i) { //from the axis //contributing to RHS PetscCall(VecSetValue(*b, m, -chargeDens * Ai * Dr_22, ADD_VALUES)); //contributing to LHS double xTheta = (Ai / Jt); for (int jt = 0; jt < Jt; ++jt) { int nj = mapIndex(k, i, jt, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &nj, &xTheta, ADD_VALUES)); } } else { //0 < i < Ir: contribution of Ai to LHS int i_jk = mapIndex(k, i - 1, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &i_jk, &Ai, ADD_VALUES)); } if (i != Ir_1) { //0 <= i < It-1: contribution of Bi to LHS int ki1j = mapIndex(k, i + 1, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &ki1j, &Bi, ADD_VALUES)); } //apply periodic BC in theta//////////////////////////// int kij_; if (!j) { kij_ = mapIndex(k, i, Jt_1, Kz, Ir); } else { kij_ = mapIndex(k, i, j - 1, Kz, Ir); } //1st contribution of Ci to LHS PetscCall(MatSetValues(*A, 1, &m, 1, &kij_, &Ci, ADD_VALUES)); int kij1; if (j == Jt_1) { kij1 = mapIndex(k, i, 0, Kz, Ir); } else { kij1 = mapIndex(k, i, j + 1, Kz, Ir); } //2nd contribution of Ci to LHS PetscCall(MatSetValues(*A, 1, &m, 1, &kij1, &Ci, ADD_VALUES)); //////////////////////////////////////////////// //apply z Neumann BC /////////////////////////////////// int k_ij; if (!k) { //z=0 k_ij = mapIndex(k, i, j, Kz, Ir); } else { //0 < z < L k_ij = mapIndex(k - 1, i, j, Kz, Ir); } PetscCall(MatSetValues(*A, 1, &m, 1, &k_ij, &Dz_2, ADD_VALUES)); ///////////////////////////////////////////////////////// //contribution of Ei to LHS int kij = mapIndex(k, i, j, Kz, Ir); PetscCall(MatSetValues(*A, 1, &m, 1, &kij, &Ei, ADD_VALUES)); } if (Nijk) { for (int i = 0; i < M; ++i) { free(Nijk[i]); } free(Nijk); } PetscCall(VecAssemblyBegin(*b)); PetscCall(VecAssemblyEnd(*b)); PetscCall(MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY)); PetscCall(MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY)); PetscFunctionReturn(PETSC_SUCCESS); } int main(int argc, char** argv) { PetscFunctionBeginUser; const char help[] = "Poisson3D solver using PETSc."; PetscCall(PetscInitialize(&argc, &argv, PETSC_NULL, help)); Mat mat_; Vec B_; Vec X_; int Nz = 26; //number of z-segments int Nr = 20; //number of r-segments int Nt = 10; //number of theta-segments PetscCall(PetscOptionsGetInt(NULL, NULL, "-z", &Nz, NULL)); PetscCall(PetscOptionsGetInt(NULL, NULL, "-r", &Nr, NULL)); PetscCall(PetscOptionsGetInt(NULL, NULL, "-t", &Nt, NULL)); //Neumann at z=0 and z=L //Dirichlet at R, the cylinder radius //On axis, compute field //number of unknowns of cylinder interior nodes minus axis int Kz = Nz - 1; int Ir = Nr - 1; int M = Kz * Ir * Nt; int bw = 15; //for 7-point stencil PetscCall(MatCreate(PETSC_COMM_WORLD, &mat_)); PetscCall(MatSetSizes(mat_, PETSC_DECIDE, PETSC_DECIDE, M, M)); PetscCall(MatSetFromOptions(mat_)); PetscCall(MatMPIAIJSetPreallocation(mat_, bw, 0, bw, 0)); PetscCall(MatSeqAIJSetPreallocation(mat_, bw, 0)); //source PetscCall(VecCreate(PETSC_COMM_WORLD, &B_)); PetscCall(VecSetSizes(B_, PETSC_DECIDE, M)); PetscCall(VecSetFromOptions(B_)); //sol PetscCall(VecDuplicate(B_, &X_)); MPIAssembler(&mat_, &B_, Kz, Ir, Nt); PCType pcType = PCEISENSTAT; KSPType solverType = KSPGMRES; int maxIters = 100; double relTol = 1.0e-6; KSP ksp; PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp)); PetscCall(KSPSetOperators(ksp, mat_, mat_)); PetscCall(KSPSetType(ksp, solverType)); //preconditioner PC pc; PetscCall(KSPGetPC(ksp, &pc)); PetscCall(PCSetType(pc, pcType)); PetscCall(KSPSetTolerances(ksp, relTol, PETSC_DEFAULT, PETSC_DEFAULT, maxIters)); PetscCall(KSPSetFromOptions(ksp)); //solving PetscCall(KSPSolve(ksp, B_, X_)); double xMin, xMax; int mnIndx, mxIndx; PetscCall(VecMin(X_, &mnIndx, &xMin)); PetscCall(VecMax(X_, &mxIndx, &xMax)); fprintf(stderr, "mnIndx=%i mxIndx=%i xMin=%g xMax=%g\n", mnIndx, mxIndx, xMin, xMax); PetscCall(PetscFinalize()); return 0; } -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From maitri.ksh at gmail.com Tue Jul 25 02:12:06 2023 From: maitri.ksh at gmail.com (maitri ksh) Date: Tue, 25 Jul 2023 10:12:06 +0300 Subject: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04) Message-ID: Hi, I am new to Petsc, here are some details of the relevant softwares I am using: 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) 2. MATLAB-R2022a 3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5') 4. configured using: './configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' I am trying to read a '.mat' file ('*myfile.mat*') using 'PetscViewerHDF5Open' (the source code '*read_Matrix.c*' is attached herewith). I could compile the source code with no error, but I am getting an error ('*ERROR.txt*') indicating that there is an issue opening the HDF5 file. Could you please help me to resolve this issue? Maitri -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- #include #include int main(int argc, char **argv) { PetscInitialize(&argc, &argv, NULL, NULL); PetscViewer viewer; Mat A; MPI_Comm comm = PETSC_COMM_WORLD; MatCreate(comm, &A); PetscObjectSetName((PetscObject)A, "A"); /* open HDF5 file for reading */ PetscViewerHDF5Open(comm, "/home/maitri/my_executables/myfile.mat", FILE_MODE_READ, &viewer); PetscViewerPushFormat(viewer, PETSC_VIEWER_HDF5_MAT); /* view matrix */ MatLoad(A, viewer); /* look up by name */ /* ... use matrix A in computation ... */ PetscViewerPopFormat(viewer); PetscViewerDestroy(&viewer); MatDestroy(&A); PetscFinalize(); return 0; } -------------- next part -------------- A non-text attachment was scrubbed... Name: myfile.mat Type: application/octet-stream Size: 386 bytes Desc: not available URL: -------------- next part -------------- maitri at LAPTOP-0CP4FI1T:~/my_executables$ ./read_Matrix HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0: #000: H5F.c line 620 in H5Fopen(): unable to open file major: File accessibility minor: Unable to open file #001: H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins major: Virtual Object Layer minor: Iteration failed #002: H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)' major: Plugin for dynamically loaded library minor: Iteration failed #003: H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin major: Plugin for dynamically loaded library minor: Can't open directory or file #004: H5VLcallback.c line 3351 in H5VL__file_open(): open failed major: Virtual Object Layer minor: Can't open object #005: H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file major: File accessibility minor: Unable to open file #006: H5Fint.c line 1990 in H5F_open(): unable to read superblock major: File accessibility minor: Read failed #007: H5Fsuper.c line 405 in H5F__super_read(): file signature not found major: File accessibility minor: Not an HDF5 file [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Fopen() Status -1 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./read_Matrix on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Tue Jul 25 09:35:41 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 [0]PETSC ERROR: #1 PetscViewerFileSetName_HDF5() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:423 [0]PETSC ERROR: #2 PetscViewerFileSetName() at /home/maitri/petsc/src/sys/classes/viewer/impls/ascii/filev.c:612 [0]PETSC ERROR: #3 PetscViewerHDF5Open() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:628 HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0: #000: H5L.c line 938 in H5Lexists(): invalid location identifier major: Invalid arguments to routine minor: Inappropriate type [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Lexists() Status -1 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./read_Matrix on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Tue Jul 25 09:35:41 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 [0]PETSC ERROR: #1 PetscViewerHDF5Traverse_Inner_Internal() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:1314 [0]PETSC ERROR: #2 PetscViewerHDF5Traverse_Internal() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:1367 [0]PETSC ERROR: #3 PetscViewerHDF5ReadAttribute() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:1236 [0]PETSC ERROR: #4 MatLoad_AIJ_HDF5() at /home/maitri/petsc/src/mat/impls/aij/seq/aijhdf5.c:63 [0]PETSC ERROR: #5 MatLoad_SeqAIJ() at /home/maitri/petsc/src/mat/impls/aij/seq/aij.c:4981 [0]PETSC ERROR: #6 MatLoad() at /home/maitri/petsc/src/mat/interface/matrix.c:1333 HDF5-DIAG: Error detected in HDF5 (1.12.2) MPI-process 0: #000: H5F.c line 668 in H5Fflush(): not a file or file object major: Invalid arguments to routine minor: Inappropriate type [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Fflush() Status -1 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./read_Matrix on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Tue Jul 25 09:35:41 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 [0]PETSC ERROR: #1 PetscViewerFlush_HDF5() at /home/maitri/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:115 [0]PETSC ERROR: #2 PetscViewerFlush() at /home/maitri/petsc/src/sys/classes/viewer/interface/flush.c:22 [0]PETSC ERROR: #3 PetscViewerDestroy() at /home/maitri/petsc/src/sys/classes/viewer/interface/view.c:94 Attempting to use an MPI routine (internal_Comm_free) before initializing or after finalizing MPICH From knepley at gmail.com Tue Jul 25 04:18:36 2023 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Jul 2023 05:18:36 -0400 Subject: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04) In-Reply-To: References: Message-ID: On Tue, Jul 25, 2023 at 3:12?AM maitri ksh wrote: > Hi, > I am new to Petsc, here are some details of the relevant softwares I am > using: > 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) > 2. MATLAB-R2022a > 3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5') > 4. configured using: './configure --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab > --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' > > I am trying to read a '.mat' file ('*myfile.mat*') using > 'PetscViewerHDF5Open' (the source code '*read_Matrix.c*' is attached > herewith). I could compile the source code with no error, but I am getting > an error ('*ERROR.txt*') indicating that there is an issue opening the > HDF5 file. Could you please help me to resolve this issue > HDF5 thinks that this is not an HDF5 file. Since PETSc built HDF5 for you, you can check this using $PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat It could be a version incompatibility between the HDF5 on your machine and the one you asked PETSc to download. Thanks, Matt > Maitri > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From samar.khatiwala at earth.ox.ac.uk Tue Jul 25 04:42:33 2023 From: samar.khatiwala at earth.ox.ac.uk (Samar Khatiwala) Date: Tue, 25 Jul 2023 09:42:33 +0000 Subject: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04) In-Reply-To: References: Message-ID: <780EE009-3A78-47F9-AFBF-79CC075A30F3@earth.ox.ac.uk> Another possibility is that the Matlab file wasn?t saved in HDF5. Try resaving it with the ?-v7.3? option. Samar On Jul 25, 2023, at 10:18 AM, Matthew Knepley wrote: On Tue, Jul 25, 2023 at 3:12?AM maitri ksh > wrote: Hi, I am new to Petsc, here are some details of the relevant softwares I am using: 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) 2. MATLAB-R2022a 3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5') 4. configured using: './configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' I am trying to read a '.mat' file ('myfile.mat') using 'PetscViewerHDF5Open' (the source code 'read_Matrix.c' is attached herewith). I could compile the source code with no error, but I am getting an error ('ERROR.txt') indicating that there is an issue opening the HDF5 file. Could you please help me to resolve this issue HDF5 thinks that this is not an HDF5 file. Since PETSc built HDF5 for you, you can check this using $PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat It could be a version incompatibility between the HDF5 on your machine and the one you asked PETSc to download. Thanks, Matt Maitri -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From maitri.ksh at gmail.com Tue Jul 25 08:40:58 2023 From: maitri.ksh at gmail.com (maitri ksh) Date: Tue, 25 Jul 2023 16:40:58 +0300 Subject: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04) In-Reply-To: References: Message-ID: I tried using '$PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat', but it gave 'h5dump error: unable to open file "/path/to/directory/myfile.mat"', *saving* the *MATLAB file* with ?*-v7.3*? option *solved* the issue. Thank you, Matthew & Samar. On Tue, Jul 25, 2023 at 12:18?PM Matthew Knepley wrote: > On Tue, Jul 25, 2023 at 3:12?AM maitri ksh wrote: > >> Hi, >> I am new to Petsc, here are some details of the relevant softwares I am >> using: >> 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) >> 2. MATLAB-R2022a >> 3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5') >> 4. configured using: './configure --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab >> --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' >> >> I am trying to read a '.mat' file ('*myfile.mat*') using >> 'PetscViewerHDF5Open' (the source code '*read_Matrix.c*' is attached >> herewith). I could compile the source code with no error, but I am getting >> an error ('*ERROR.txt*') indicating that there is an issue opening the >> HDF5 file. Could you please help me to resolve this issue >> > > HDF5 thinks that this is not an HDF5 file. Since PETSc built HDF5 for you, > you can check this using > > $PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat > > It could be a version incompatibility between the HDF5 on your machine and > the one you asked PETSc to download. > > Thanks, > > Matt > > >> Maitri >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yixuan.sun at anl.gov Tue Jul 25 13:41:55 2023 From: yixuan.sun at anl.gov (Sun, Yixuan) Date: Tue, 25 Jul 2023 18:41:55 +0000 Subject: [petsc-users] Question about using PETSc to generate matrix preconditioners Message-ID: Hello PETSc team, My name is Yixuan, and I am a postdoc at MCS. We are working on a project where we want to predict the preconditioners from the corresponding matrices. For preliminary exploration purposes, we generated random matrices and their preconditioners using PETSc. The current deep learning models look promising with the generated data. However, we are not sure about the correctness of the data generation code and wanted to ask if you could help check the correctness of our script (< 100 lines) and let us know if there were any issues. Here is the link to the script. Please let me know if you have any questions or concerns. Thank you in advance! Warm regards, Yixuan -------------------- Yixuan Sun Postdoctoral Researcher Mathematics and Computer Science Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 25 15:15:24 2023 From: jed at jedbrown.org (Jed Brown) Date: Tue, 25 Jul 2023 12:15:24 -0800 Subject: [petsc-users] Question about using PETSc to generate matrix preconditioners In-Reply-To: References: Message-ID: <87o7jzagtf.fsf@jedbrown.org> I think random matrices will produce misleading results. The chance of randomly generating a matrix that resembles an application is effectively zero. I think you'd be better off with some model problems varying parameters that control the physical regime (e.g., shifts to a Laplacian, advection with/without upwinding at varying cell Peclet numbers, varying nondimensional parameters, time step, and boundary layers for fluids, varying Poisson ratio and material contrast in elasticity). "Sun, Yixuan via petsc-users" writes: > Hello PETSc team, > > My name is Yixuan, and I am a postdoc at MCS. We are working on a project where we want to predict the preconditioners from the corresponding matrices. For preliminary exploration purposes, we generated random matrices and their preconditioners using PETSc. The current deep learning models look promising with the generated data. However, we are not sure about the correctness of the data generation code and wanted to ask if you could help check the correctness of our script (< 100 lines) and let us know if there were any issues. Here is the link to the script. > > Please let me know if you have any questions or concerns. Thank you in advance! > > > Warm regards, > Yixuan > -------------------- > Yixuan Sun > Postdoctoral Researcher > Mathematics and Computer Science > Argonne National Laboratory From maitri.ksh at gmail.com Tue Jul 25 17:06:39 2023 From: maitri.ksh at gmail.com (maitri ksh) Date: Wed, 26 Jul 2023 01:06:39 +0300 Subject: [petsc-users] Null Pointer Issue (Eigen-Value-Problem with user input Matrices) Message-ID: I am using the following setup: 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) 2. MATLAB-R2022a 3. slepc-3.19.1 4. configured using: './configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' I am trying to solve an eigenvalue problem with user input matrices 'A.petsc' & 'B.petsc' using '*solveEigValProb.c*', there was no issue during compiling but when I try to run the executable it produces error (' *ERROR.txt*') indicating a null pointer issue but I am not sure what the issue is. Maitri -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- #include #include #include int main(int argc, char **argv) { PetscInitialize(&argc, &argv, NULL, NULL); /*PetscViewer viewer;*/ Mat A, B; EPS eps; MPI_Comm comm = PETSC_COMM_WORLD; PetscInt nev = 2; // Number of eigenvalues to compute PetscScalar sigma = 0.0; // Shift value MatCreate(comm, &A); PetscObjectSetName((PetscObject)A, "A"); /* open file for reading */ PetscViewer viewerA; PetscErrorCode ierr; ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "A.petsc", FILE_MODE_READ, &viewerA);CHKERRQ(ierr); ierr = MatLoad(A, viewerA);CHKERRQ(ierr); PetscViewerDestroy(&viewerA); /* open file for reading */ PetscViewer viewerB; ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "B.petsc", FILE_MODE_READ, &viewerB);CHKERRQ(ierr); ierr = MatLoad(B, viewerB);CHKERRQ(ierr); PetscViewerDestroy(&viewerB); /* ... use matrices in computation ... */ // Create the eigensolver context ierr = EPSCreate(PETSC_COMM_WORLD, &eps); CHKERRQ(ierr); // Set operators A and B ierr = EPSSetOperators(eps, A, B); CHKERRQ(ierr); // Set dimensions and target (if needed) ierr = EPSSetDimensions(eps, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); ierr = EPSSetTarget(eps, sigma); CHKERRQ(ierr); // Solve the eigenvalue problem ierr = EPSSolve(eps); CHKERRQ(ierr); // Get the number of computed eigenvalues ierr = EPSGetConverged(eps, &nev); CHKERRQ(ierr); // Get the computed eigenvalues and eigenvectors PetscScalar eigenvalue, kr; Vec eigenvector; Vec xr, xi; PetscPrintf(PETSC_COMM_WORLD, "Eigenvalues:\n"); for (PetscInt i = 0; i < nev; i++) { ierr = EPSGetEigenpair(eps, i, &eigenvalue, &kr, xr, xi); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD, "Eigenvalue %d: %.8f + %.8fi\n", i, PetscRealPart(eigenvalue), PetscImaginaryPart(eigenvalue)); } // Destroy everything ierr = EPSDestroy(&eps); CHKERRQ(ierr); ierr = MatDestroy(&A); CHKERRQ(ierr); ierr = MatDestroy(&B); CHKERRQ(ierr); PetscFinalize(); return 0; } -------------- next part -------------- maitri at LAPTOP-0CP4FI1T:~/my_executables$ ./new2 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Pointer: Parameter # 1 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./new2 on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Wed Jul 26 00:54:45 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 [0]PETSC ERROR: #1 MatLoad() at /home/maitri/petsc/src/mat/interface/matrix.c:1317 [0]PETSC ERROR: #2 main() at new2.c:30 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_SELF, 85) - process 0 [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=85 message=application called MPI_Abort(MPI_COMM_SELF, 85) - process 0 : system msg for write_line failure : Bad file descriptor From bsmith at petsc.dev Tue Jul 25 17:13:42 2023 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 25 Jul 2023 18:13:42 -0400 Subject: [petsc-users] Null Pointer Issue (Eigen-Value-Problem with user input Matrices) In-Reply-To: References: Message-ID: <8AF9E884-B47C-47D5-B745-D78A1AB2AC30@petsc.dev> You never called MatCreate(comm, &B); > On Jul 25, 2023, at 6:06 PM, maitri ksh wrote: > > I am using the following setup: > 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform) > 2. MATLAB-R2022a > 3. slepc-3.19.1 > 4. configured using: './configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1' > > I am trying to solve an eigenvalue problem with user input matrices 'A.petsc' & 'B.petsc' using 'solveEigValProb.c', there was no issue during compiling but when I try to run the executable it produces error ('ERROR.txt') indicating a null pointer issue but I am not sure what the issue is. > > Maitri > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maitri.ksh at gmail.com Wed Jul 26 07:12:46 2023 From: maitri.ksh at gmail.com (maitri ksh) Date: Wed, 26 Jul 2023 15:12:46 +0300 Subject: [petsc-users] Inverse of a Large Sparse Matrix Message-ID: I have a large sparse matrix (480000x480000) and I need to take its inverse and use it to solve an eigenvalue problem. According to petsc-archive , Barry suggested using superLU with MatMatSolve() (& not KSPsolver) for matrix sizes of 5000 to 20000. I was wondering two things: a) is it possible to avoid taking the explicit inverse of the large sparse matrix (as was discussed in the archive) for this particular case (in which I am using the matrix-inverse for an eigenvalue problem) b) is KSPsolver more suitable here? Also, can someone please explain why SuperLU (SuperLU_SEQUENTIAL, which does not involve any parallel computation) is more efficient in dealing with large sparse matrices as compared to MATLAB's inbuilt LU solver. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 26 07:28:50 2023 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Jul 2023 08:28:50 -0400 Subject: [petsc-users] Inverse of a Large Sparse Matrix In-Reply-To: References: Message-ID: On Wed, Jul 26, 2023 at 8:13?AM maitri ksh wrote: > I have a large sparse matrix (480000x480000) and I need to take its > inverse and use it to solve an eigenvalue problem. According to > petsc-archive > , > Barry suggested using superLU with MatMatSolve() (& not KSPsolver) for > matrix sizes of 5000 to 20000. I was wondering two things: > a) is it possible to avoid taking the explicit inverse of the large sparse > matrix (as was discussed in the archive) for this particular case (in which > I am using the matrix-inverse for an eigenvalue problem) > You do not actually want to explicitly invert in most instances. SLEPc will do the right thing automatically. > b) is KSPsolver more suitable here? > Most likely. > Also, can someone please explain why SuperLU (SuperLU_SEQUENTIAL, which > does not involve any parallel computation) is more efficient in dealing > with large sparse matrices as compared to MATLAB's inbuilt LU > solver. > I have no idea what MATLAB is doing. However SuperLU uses the supernodal formulation, and I am not sure that MATLAB does. If you care about the last ounce of performance, it is always worth trying several packages, so you might compare with the serial PETSc LU and MUMPS. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.wick.1980 at gmail.com Wed Jul 26 20:47:23 2023 From: michael.wick.1980 at gmail.com (Michael Wick) Date: Thu, 27 Jul 2023 09:47:23 +0800 Subject: [petsc-users] Monolithic AMG with fieldsplit as smoother Message-ID: Hello PETSc team: I wonder if the current PETSc implementation supports using AMG monolithically for a multi-field problem and using fieldsplit in the smoother. Thank you very much, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 26 21:34:03 2023 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 26 Jul 2023 22:34:03 -0400 Subject: [petsc-users] [petsc-maint] Monolithic AMG with fieldsplit as smoother In-Reply-To: References: Message-ID: <00278050-C186-4D94-A055-A6AFC8777891@petsc.dev> See the very end of the section https://petsc.org/release/manual/ksp/#multigrid-preconditioners on how to control the smoothers (and coarse grid solve) for multigrid in PETSc including for algebraic multigrid. So, for example, -mg_levels_pc_type fieldsplit would be the starting point. Depending on the block size of the matrices it may automatically do simple splits, you can control the details of the fieldsplit preconditioner with -mg_levels_pc_fieldsplit_... and the details for each split with -mg_levels_fieldsplit_.... See src/ksp/ksp/tutorials/ex42.c for example, usage Feel free to ask more specific questions once you get started. > On Jul 26, 2023, at 9:47 PM, Michael Wick wrote: > > Hello PETSc team: > > I wonder if the current PETSc implementation supports using AMG monolithically for a multi-field problem and using fieldsplit in the smoother. > > Thank you very much, > > Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Jul 26 23:48:13 2023 From: jed at jedbrown.org (Jed Brown) Date: Wed, 26 Jul 2023 20:48:13 -0800 Subject: [petsc-users] [petsc-maint] Monolithic AMG with fieldsplit as smoother In-Reply-To: <00278050-C186-4D94-A055-A6AFC8777891@petsc.dev> References: <00278050-C186-4D94-A055-A6AFC8777891@petsc.dev> Message-ID: <87r0ou7yeq.fsf@jedbrown.org> AMG is subtle here. With AMG for systems, you typically feed it elements of the near null space. In the case of (smoothed) aggregation, the coarse space will have a regular block structure with block sizes equal to the number of near-null vectors. You can use pc_fieldsplit options to select which fields you want in each split. However, AMG also needs a strength of connection and if your system is so weird you need to fieldsplit the smoothers (e.g., a saddle point problem or a hyperbolic system) then it's likely that you'll also need a custom strength of connection to obtain reasonable coarsening. Barry Smith writes: > See the very end of the section https://petsc.org/release/manual/ksp/#multigrid-preconditioners on how to control the smoothers (and coarse grid solve) for multigrid in PETSc including for algebraic multigrid. > > So, for example, -mg_levels_pc_type fieldsplit would be the starting point. Depending on the block size of the matrices it may automatically do simple splits, you can control the details of the fieldsplit preconditioner with -mg_levels_pc_fieldsplit_... and the details for each split with -mg_levels_fieldsplit_.... > > See src/ksp/ksp/tutorials/ex42.c for example, usage > > Feel free to ask more specific questions once you get started. > >> On Jul 26, 2023, at 9:47 PM, Michael Wick wrote: >> >> Hello PETSc team: >> >> I wonder if the current PETSc implementation supports using AMG monolithically for a multi-field problem and using fieldsplit in the smoother. >> >> Thank you very much, >> >> Mike From knepley at gmail.com Thu Jul 27 00:09:32 2023 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Jul 2023 01:09:32 -0400 Subject: [petsc-users] [petsc-maint] Monolithic AMG with fieldsplit as smoother In-Reply-To: <87r0ou7yeq.fsf@jedbrown.org> References: <00278050-C186-4D94-A055-A6AFC8777891@petsc.dev> <87r0ou7yeq.fsf@jedbrown.org> Message-ID: On Thu, Jul 27, 2023 at 12:48?AM Jed Brown wrote: > AMG is subtle here. With AMG for systems, you typically feed it elements > of the near null space. In the case of (smoothed) aggregation, the coarse > space will have a regular block structure with block sizes equal to the > number of near-null vectors. You can use pc_fieldsplit options to select > which fields you want in each split. > > However, AMG also needs a strength of connection and if your system is so > weird you need to fieldsplit the smoothers (e.g., a saddle point problem or > a hyperbolic system) then it's likely that you'll also need a custom > strength of connection to obtain reasonable coarsening. > For this reason, sometimes GMG is easier for systems since you just rediscretize. Thanks, Matt > Barry Smith writes: > > > See the very end of the section > https://petsc.org/release/manual/ksp/#multigrid-preconditioners on how to > control the smoothers (and coarse grid solve) for multigrid in PETSc > including for algebraic multigrid. > > > > So, for example, -mg_levels_pc_type fieldsplit would be the starting > point. Depending on the block size of the matrices it may automatically do > simple splits, you can control the details of the fieldsplit > preconditioner with -mg_levels_pc_fieldsplit_... and the details for each > split with -mg_levels_fieldsplit_.... > > > > See src/ksp/ksp/tutorials/ex42.c for example, usage > > > > Feel free to ask more specific questions once you get started. > > > >> On Jul 26, 2023, at 9:47 PM, Michael Wick > wrote: > >> > >> Hello PETSc team: > >> > >> I wonder if the current PETSc implementation supports using AMG > monolithically for a multi-field problem and using fieldsplit in the > smoother. > >> > >> Thank you very much, > >> > >> Mike > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jul 27 08:50:19 2023 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 27 Jul 2023 09:50:19 -0400 Subject: [petsc-users] [petsc-maint] Monolithic AMG with fieldsplit as smoother In-Reply-To: References: <00278050-C186-4D94-A055-A6AFC8777891@petsc.dev> <87r0ou7yeq.fsf@jedbrown.org> Message-ID: I would not worry about the null space (if you have elasticity or the equivalent use hypre for now) and the strength of connections, is not very useful in my experience (it is confounded my high order and no one has bothered to deploy a fancy strength of connections method in a library that I know of). If you have anisotropies or material discontinuities, honestly AMG does not do as well as advertised. That said, we could talk after you get up and running. If your problems are very hard then as Matt said, old fashioned geometric MG using modern unstructured (FE) discretizations and mesh management, is something to consider. PETSc has support for this and we are actively using and developing support for this. Antony Jameson has been doing this for decades here is an example of a new project doing something like this: https://arxiv.org/abs/2307.04528. Tobin Isaac, in PETSc, and many others have done things like this, but they tend to be customized for an application whereas AMG strives to be general. Mark On Thu, Jul 27, 2023 at 1:10?AM Matthew Knepley wrote: > On Thu, Jul 27, 2023 at 12:48?AM Jed Brown wrote: > >> AMG is subtle here. With AMG for systems, you typically feed it elements >> of the near null space. In the case of (smoothed) aggregation, the coarse >> space will have a regular block structure with block sizes equal to the >> number of near-null vectors. You can use pc_fieldsplit options to select >> which fields you want in each split. >> >> However, AMG also needs a strength of connection and if your system is so >> weird you need to fieldsplit the smoothers (e.g., a saddle point problem or >> a hyperbolic system) then it's likely that you'll also need a custom >> strength of connection to obtain reasonable coarsening. >> > > For this reason, sometimes GMG is easier for systems since you just > rediscretize. > > Thanks, > > Matt > > >> Barry Smith writes: >> >> > See the very end of the section >> https://petsc.org/release/manual/ksp/#multigrid-preconditioners on how >> to control the smoothers (and coarse grid solve) for multigrid in PETSc >> including for algebraic multigrid. >> > >> > So, for example, -mg_levels_pc_type fieldsplit would be the starting >> point. Depending on the block size of the matrices it may automatically do >> simple splits, you can control the details of the fieldsplit >> preconditioner with -mg_levels_pc_fieldsplit_... and the details for each >> split with -mg_levels_fieldsplit_.... >> > >> > See src/ksp/ksp/tutorials/ex42.c for example, usage >> > >> > Feel free to ask more specific questions once you get started. >> > >> >> On Jul 26, 2023, at 9:47 PM, Michael Wick >> wrote: >> >> >> >> Hello PETSc team: >> >> >> >> I wonder if the current PETSc implementation supports using AMG >> monolithically for a multi-field problem and using fieldsplit in the >> smoother. >> >> >> >> Thank you very much, >> >> >> >> Mike >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maitri.ksh at gmail.com Thu Jul 27 08:50:33 2023 From: maitri.ksh at gmail.com (maitri ksh) Date: Thu, 27 Jul 2023 16:50:33 +0300 Subject: [petsc-users] MUMPS Error 'INFOG(1)=-3 INFO(2)=3' (SPARSE MATRIX INVERSE) Message-ID: I am using 'MatMumpsGetInverse()' to get the inverse of a sparse matrix. I am using parts of ex214.c code to get the inverse, but I get an error that seems to be coming from MUMPS-library. Any suggestions? ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error reported by MUMPS in solve phase: INFOG(1)=-3 INFO(2)=3 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./MatInv_MUMPS on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Thu Jul 27 16:35:02 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-bison --download-cmake [0]PETSC ERROR: #1 MatMumpsGetInverse_MUMPS() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2720 [0]PETSC ERROR: #2 MatMumpsGetInverse() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2753 [0]PETSC ERROR: #3 main() at MatInv_MUMPS.c:74 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=76 message=application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 : system msg for write_line failure : Bad file descriptor Maitri -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- maitri at LAPTOP-0CP4FI1T:~/my_executables$ ./MatInv_MUMPS using LU factorization [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error reported by MUMPS in solve phase: INFOG(1)=-3 INFO(2)=3 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown [0]PETSC ERROR: ./MatInv_MUMPS on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Thu Jul 27 16:35:02 2023 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-bison --download-cmake [0]PETSC ERROR: #1 MatMumpsGetInverse_MUMPS() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2720 [0]PETSC ERROR: #2 MatMumpsGetInverse() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2753 [0]PETSC ERROR: #3 main() at MatInv_MUMPS.c:74 [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=76 message=application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 : system msg for write_line failure : Bad file descriptor -------------- next part -------------- #include #include int main(int argc,char **args) { PetscErrorCode ierr; PetscMPIInt size,rank; Mat A,F,X,spRHST; PetscInt m,n,nrhs,M,N,i,test; PetscScalar v; PetscReal norm,tol=PETSC_SQRT_MACHINE_EPSILON; PetscRandom rand; PetscBool displ=PETSC_FALSE; ierr = PetscInitialize(&argc, &args, NULL, NULL);if (ierr) return ierr; ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = PetscOptionsGetBool(NULL,NULL,"-displ",&displ,NULL);CHKERRQ(ierr); /* Load matrix A */ PetscViewer viewerA; ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "A1.petsc", FILE_MODE_READ, &viewerA);CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD, &A);CHKERRQ(ierr); ierr = MatLoad(A, viewerA);CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewerA);CHKERRQ(ierr); ierr = MatGetLocalSize(A,&m,&n);CHKERRQ(ierr); ierr = MatGetSize(A,&M,&N);CHKERRQ(ierr); if (m != n) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ, "The matrix is not square (%d, %d)", m, n); /* Create dense matrix X */ nrhs = N; ierr = PetscOptionsGetInt(NULL,NULL,"-nrhs",&nrhs,NULL);CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD, &X);CHKERRQ(ierr); ierr = MatSetSizes(X, m, PETSC_DECIDE, PETSC_DECIDE, nrhs);CHKERRQ(ierr); ierr = MatSetType(X, MATDENSE);CHKERRQ(ierr); ierr = MatSetFromOptions(X);CHKERRQ(ierr); ierr = MatSetUp(X);CHKERRQ(ierr); ierr = PetscRandomCreate(PETSC_COMM_WORLD,&rand);CHKERRQ(ierr); ierr = PetscRandomSetFromOptions(rand);CHKERRQ(ierr); ierr = MatSetRandom(X,rand);CHKERRQ(ierr); // factorise 'A' using LU Factorization ierr = PetscPrintf(PETSC_COMM_WORLD,"using LU factorization\n");CHKERRQ(ierr); ierr = MatGetFactor(A,MATSOLVERMUMPS,MAT_FACTOR_LU,&F);CHKERRQ(ierr); ierr = MatLUFactorSymbolic(F,A,NULL,NULL,NULL);CHKERRQ(ierr); ierr = MatLUFactorNumeric(F,A,NULL);CHKERRQ(ierr); // Create spRHST: PETSc does not support compressed column format which is required by MUMPS for sparse RHS matrix, // thus user must create spRHST=spRHS^T ierr = MatCreate(PETSC_COMM_WORLD,&spRHST);CHKERRQ(ierr); if (!rank) { /* MUMPS requires RHS be centralized on the host! */ ierr = MatSetSizes(spRHST,nrhs,M,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); } else { ierr = MatSetSizes(spRHST,0,0,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); } ierr = MatSetType(spRHST,MATAIJ);CHKERRQ(ierr); ierr = MatSetFromOptions(spRHST);CHKERRQ(ierr); ierr = MatSetUp(spRHST);CHKERRQ(ierr); if (!rank) { v = 1.0; for (i=0; i References: Message-ID: <712F1E8A-B69C-45B3-9EBC-99B3F68B65B2@joliv.et> MUMPS errors are documented in section 8 of https://mumps-solver.org/doc/userguide_5.6.1.pdf Thanks, Pierre > On 27 Jul 2023, at 3:50 PM, maitri ksh wrote: > > I am using 'MatMumpsGetInverse()' to get the inverse of a sparse matrix. I am using parts of ex214.c code to get the inverse, but I get an error that seems to be coming from MUMPS-library. Any suggestions? > > ERROR: > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error reported by MUMPS in solve phase: INFOG(1)=-3 INFO(2)=3 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown > [0]PETSC ERROR: ./MatInv_MUMPS on a arch-linux-c-debug named LAPTOP-0CP4FI1T by maitri Thu Jul 27 16:35:02 2023 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1 --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-bison --download-cmake > [0]PETSC ERROR: #1 MatMumpsGetInverse_MUMPS() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2720 > [0]PETSC ERROR: #2 MatMumpsGetInverse() at /home/maitri/petsc/src/mat/impls/aij/mpi/mumps/mumps.c:2753 > [0]PETSC ERROR: #3 main() at MatInv_MUMPS.c:74 > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 > [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=76 message=application called MPI_Abort(MPI_COMM_SELF, 76) - process 0 > : > system msg for write_line failure : Bad file descriptor > > > Maitri > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Fri Jul 28 14:59:40 2023 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Fri, 28 Jul 2023 12:59:40 -0700 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Fri Jul 28 19:58:04 2023 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Fri, 28 Jul 2023 17:58:04 -0700 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: References: Message-ID: <76B6A62E-7D56-4C7F-9302-BDEF6062931B@gmail.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cover.jpg Type: image/png Size: 139576 bytes Desc: not available URL: From jed at jedbrown.org Fri Jul 28 21:04:30 2023 From: jed at jedbrown.org (Jed Brown) Date: Fri, 28 Jul 2023 20:04:30 -0600 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: <76B6A62E-7D56-4C7F-9302-BDEF6062931B@gmail.com> References: <76B6A62E-7D56-4C7F-9302-BDEF6062931B@gmail.com> Message-ID: <87cz0b8ocx.fsf@jedbrown.org> See src/snes/tutorials/ex70.c for the code that I think was used for that paper. Alexander Lindsay writes: > Sorry for the spam. Looks like these authors have published multiple papers on the subject > > cover.jpg > Combining the Augmented Lagrangian Preconditioner with the Simple Schur Complement Approximation | SIAM Journal on > Scientific Computingdoi.org > cover.jpg > > On Jul 28, 2023, at 12:59 PM, Alexander Lindsay wrote: > > ?Do you know of anyone who has applied the augmented Lagrange methodology to a finite volume discretization? > > On Jul 6, 2023, at 6:25 PM, Matthew Knepley wrote: > > ?On Thu, Jul 6, 2023 at 8:30?PM Alexander Lindsay wrote: > > This is an interesting article that compares a multi-level ILU algorithm to approximate commutator and augmented > Lagrange methods: https://doi.org/10.1002/fld.5039 > > That is for incompressible NS. The results are not better than https://arxiv.org/abs/1810.03315, and that PC is considerably > simpler and already implemented in PETSc. There is an update in to this > > > https://epubs.siam.org/doi/abs/10.1137/21M1430698?casa_token=Fp_XhuZStZ0AAAAA:YDhnkW9XvAom_b8KocWz-hBEI7FAt46aw3ICa0FvCrOVCtYr9bwvtqJ4aBOTkDSvANKh6YTQEw > > > which removes the need for complicated elements. > > You might need stuff like ILU for compressible flow, but I think incompressible is solved. > > Thanks, > > Matt > > On Wed, Jun 28, 2023 at 11:37?AM Alexander Lindsay wrote: > > I do believe that based off the results in https://doi.org/10.1137/040608817 we should be able to make LSC, with > proper scaling, compare very favorably with PCD > > On Tue, Jun 27, 2023 at 10:41?AM Alexander Lindsay wrote: > > I've opened https://gitlab.com/petsc/petsc/-/merge_requests/6642 which adds a couple more scaling > applications of the inverse of the diagonal of A > > On Mon, Jun 26, 2023 at 6:06?PM Alexander Lindsay wrote: > > I guess that similar to the discussions about selfp, the approximation of the velocity mass matrix by the > diagonal of the velocity sub-matrix will improve when running a transient as opposed to a steady > calculation, especially if the time derivative is lumped.... Just thinking while typing > > On Mon, Jun 26, 2023 at 6:03?PM Alexander Lindsay wrote: > > Returning to Sebastian's question about the correctness of the current LSC implementation: in the > taxonomy paper that Jed linked to (which talks about SIMPLE, PCD, and LSC), equation 21 shows four > applications of the inverse of the velocity mass matrix. In the PETSc implementation there are at > most two applications of the reciprocal of the diagonal of A (an approximation to the velocity mass > matrix without more plumbing, as already pointed out). It seems like for code implementations in > which there are possible scaling differences between the velocity and pressure equations, that this > difference in the number of inverse applications could be significant? I know Jed said that these > scalings wouldn't really matter if you have a uniform grid, but I'm not 100% convinced yet. > > I might try fiddling around with adding two more reciprocal applications. > > On Fri, Jun 23, 2023 at 1:09?PM Pierre Jolivet wrote: > > On 23 Jun 2023, at 10:06 PM, Pierre Jolivet wrote: > > On 23 Jun 2023, at 9:39 PM, Alexander Lindsay wrote: > > Ah, I see that if I use Pierre's new 'full' option for -mat_schur_complement_ainv_type > > That was not initially done by me > > Oops, sorry for the noise, looks like it was done by me indeed in > 9399e4fd88c6621aad8fe9558ce84df37bd6fada? > > Thanks, > Pierre > > (though I recently tweaked MatSchurComplementComputeExplicitOperator() a bit to use > KSPMatSolve(), so that if you have a small Schur complement ? which is not really the case > for NS ? this could be a viable option, it was previously painfully slow). > > Thanks, > Pierre > > that I get a single iteration for the Schur complement solve with LU. That's a nice testing > option > > On Fri, Jun 23, 2023 at 12:02?PM Alexander Lindsay > wrote: > > I guess it is because the inverse of the diagonal form of A00 becomes a poor > representation of the inverse of A00? I guess naively I would have thought that the > blockdiag form of A00 is A00 > > On Fri, Jun 23, 2023 at 10:18?AM Alexander Lindsay > wrote: > > Hi Jed, I will come back with answers to all of your questions at some point. I > mostly just deal with MOOSE users who come to me and tell me their solve is > converging slowly, asking me how to fix it. So I generally assume they have > built an appropriate mesh and problem size for the problem they want to solve > and added appropriate turbulence modeling (although my general assumption > is often violated). > > > And to confirm, are you doing a nonlinearly implicit velocity-pressure solve? > > Yes, this is our default. > > A general question: it seems that it is well known that the quality of selfp > degrades with increasing advection. Why is that? > > On Wed, Jun 7, 2023 at 8:01?PM Jed Brown wrote: > > Alexander Lindsay writes: > > > This has been a great discussion to follow. Regarding > > > >> when time stepping, you have enough mass matrix that cheaper > preconditioners are good enough > > > > I'm curious what some algebraic recommendations might be for high Re > in > > transients. > > What mesh aspect ratio and streamline CFL number? Assuming your model > is turbulent, can you say anything about momentum thickness Reynolds > number Re_?? What is your wall normal spacing in plus units? (Wall > resolved or wall modeled?) > > And to confirm, are you doing a nonlinearly implicit velocity-pressure > solve? > > > I've found one-level DD to be ineffective when applied monolithically or > to the momentum block of a split, as it scales with the mesh size. > > I wouldn't put too much weight on "scaling with mesh size" per se. You > want an efficient solver for the coarsest mesh that delivers sufficient > accuracy in your flow regime. Constants matter. > > Refining the mesh while holding time steps constant changes the advective > CFL number as well as cell Peclet/cell Reynolds numbers. A meaningful > scaling study is to increase Reynolds number (e.g., by growing the domain) > while keeping mesh size matched in terms of plus units in the viscous > sublayer and Kolmogorov length in the outer boundary layer. That turns > out to not be a very automatic study to do, but it's what matters and you > can spend a lot of time chasing ghosts with naive scaling studies. > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any > results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From bsmith at petsc.dev Sat Jul 29 10:06:45 2023 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 29 Jul 2023 11:06:45 -0400 Subject: [petsc-users] PCMG with PCREDISTRIBUTE In-Reply-To: References: <82DBCDB1-99C2-40A4-9741-D348AC5D5B3A@petsc.dev> <044DFE3A-95D6-48AA-B6DA-FBB228975597@unipv.it> <4347EFD4-D04E-4DEB-8313-B313AF5F4E02@unipv.it> <66740C62-780A-41C5-9228-1B91FE8D4115@petsc.dev> <7FDF0ACC-CBF1-427D-B1F2-425D8877DB9D@petsc.dev> <42623281-600D-4D09-9553-6A4927640266@petsc.dev> Message-ID: <1ABEB352-E6CE-461F-A1D6-6CD67D028C8B@petsc.dev> You may be able to use https://petsc.org/main/manualpages/PC/PCSetApplicationContext/#pcsetapplicationcontext or KSPSetApplicationContext() or PetscObjectCompose() (this may be better since it can be eventually merged into PETSc). Barry > On Jul 29, 2023, at 7:54 AM, Carl-Johan Thore wrote: > > Hi again, > > I?ve now managed to get PCMG with PCREDISTRIBUTE to work on multiple ranks, and convergence looks good on my > examples. (the modification to redistribute.c was trivial ? just skip the case ?if (size > 1) {? in PCSetUp_Redistribute). > > The final step is your suggestion > > ?2) the only problem with 1 is it is likely to be poorly load balanced (but you can make some runs to see how imbalanced it is, that will depend exactly on what parts are locked and what MPI processes they are on). So if it is poorly balanced then you would need to get out of redistribute.c a mapping for each kept dof to what MPI rank it is moved to and use that to move the entries in the reduced interpolation you have created. ? > > My idea for this is to do ?VecScatterBegin(red->scatter ?? on my vector indicating fixed or free DOFs to redistribute it in the same way as the global matrix is redistributed. > It is tempting to do something like ?KSPGetPC(ksp, &pc); PC_Redistribute *red = (PC_Redistribute *)pc->data;? in my main code, and then do the scatter with the red object. But this is not directly > possible, resulting in PC_Redistribute not defined and so on. Another alternative is perhaps to pass my vector into the PC via the ?optional user-defined context? in the _p_PC context. > Do you have any suggestions or hints for a good (efficient and non-intrusive) way to ?get out of redistribute.c a mapping ??? > > /Carl-Johan > > > From: Carl-Johan Thore > > Sent: Tuesday, July 25, 2023 4:49 PM > To: Barry Smith > > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > Ok thanks! Good to know I'm not doing anything wrong on the PETSc-side, so to speak. It seems like it could be related to WSL then > From: Barry Smith > > Sent: 25 July 2023 16:39:37 > To: Carl-Johan Thore > > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > No idea why it would be particularly slow. Probably some issue with making the PETSc shared library, normally that would take a few seconds on Linux or a Mac. > > > > > On Jul 25, 2023, at 6:12 AM, Carl-Johan Thore > wrote: > > Hi again, > > I?ve now gotten PCMG with PCREDISTRIBUTE to work nicely on simple examples on one core > (attached ppt). I therefore plan to test with multiple cores by first modifying redistribute.c such > that the dofs are not moved around as you suggested. For this I wonder what is the recommended way > of modifying a single file in my PETSc-installation? I?ve tried running make in the folder containing redistribute.c: > > > > This works but is terribly slow on my computer (google indicates that this could be related > to me using Windows Subsystem for Linux, but I didn?t succeed with the suggestions) > > Kind regards, > Carl-Johan > > > > From: Carl-Johan Thore > > Sent: Sunday, July 2, 2023 8:07 PM > To: Barry Smith > > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > "NN" is used to construct the row indices passed to matzerorowscolumns. That works fine, so at least in that sense NN is correct. But I should probably compare my IS:es with those row indices as well. > > Yes, making a tiny problem is a good idea. > > Thanks for taking time to look at this > From: Barry Smith > > Sent: 02 July 2023 18:58:50 > To: Carl-Johan Thore > > Cc: petsc-users > > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > Are you sure the NN is correct? I cannot see how you set that and know that it exactly matches the way PCREDISTRIBUTE selects rows? > > I suggest making a tiny problem with artificial matrix values that you select to "slice" of parts of the grid, so you can see exactly on the grid that the selected rows and columns are correct as you expect. > > > On Jul 2, 2023, at 2:16 AM, Carl-Johan Thore > wrote: > > Hi, > > I tried your suggestion > > ?1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good as on the full system.? > > from below and got the size of the interpolation matrix correct. But the convergence doesn?t seem right. In the attached .txt-file the code without redistribute convergences in 8 > FGMRES iterations whereas with redistribute it takes 25 (I?ve tested this on various meshes and the redistribute-code consistently performs much worse in terms of number of iterations). The code without redistribute is very well tested and always performs very well, so I?m fairly certain the error is in my new code. > > Would you be willing to take a quick look at the attached code snippet to see if I?m doing some obvious mistake? > > Kind regards, > Carl-Johan > > From: Barry Smith > > Sent: Friday, June 30, 2023 5:21 PM > To: Matthew Knepley > > Cc: Carl-Johan Thore >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > > > On Jun 30, 2023, at 10:22 AM, Matthew Knepley > wrote: > > On Fri, Jun 30, 2023 at 10:16?AM Carl-Johan Thore via petsc-users > wrote: > Thanks for the quick reply and the suggestions! > > ? ? you should first check that the PCMG works quite well ? > > Yes, the PCMG works very well for the full system. > > ?I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1.? > > Yes, it?s slightly different so problem 2 should be solved. > > It looked somewhat complicated to get PCMG to work with redistribute, so I?ll try with PCGAMG first > (it ran immediately with redistribute, but was slower than PCMG on my, very small, test problem. I?ll try > to tune the settings). > > A related question: I?m here using a DMDA for a structured grid but I?m locking so many DOFs that for many of the elements > all DOFs are locked. In such a case could it make sense to switch/convert the DMDA to a DMPlex containing only those > elements that actually have DOFs? > > Possibly, but if you are doing FD, then there is built-in topology in DMDA that is not present in Plex, so > finding the neighbors in the right order is harder (possible, but harder, we address this in some new work that is not yet merged). There is also structured adaptive support with DMForest, but this also does not preserve the stencil. > > The efficiency of active set VI solvers in PETSc demonstrates to me that solving reduced systems can be done efficiently with geometric multigrid using a structured grid so I would not suggest giving up on what you started. > > You can do it in two steps > > 1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good as on the full system. > > 2) the only problem with 1 is it is likely to be poorly load balanced (but you can make some runs to see how imbalanced it is, that will depend exactly on what parts are locked and what MPI processes they are on). So if it is poorly balanced then you would need to get out of redistribute.c a mapping for each kept dof to what MPI rank it is moved to and use that to move the entries in the reduced interpolation you have created. > > If you do succeed it would actually be useful code that we could add to PCREDISTRIBUTE for more general use by others. > > Barry > > > > > > Thanks, > > Matt > > From: Barry Smith > > Sent: Friday, June 30, 2023 3:57 PM > To: Carl-Johan Thore > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > Oh, I forgot to mention you should first check that the PCMG works quite well for the full system (without the PCREDISTRIBUTE); the convergence > on the redistributed system (assuming you did all the work to get PCMG to work for you) should be very similar to (but not measurably better) than the convergence on the full system. > > > > > On Jun 30, 2023, at 9:17 AM, Barry Smith > wrote: > > > ex42.c provides directly the interpolation/restriction needed to move between levels in the loop > > for (k = 1; k < nlevels; k++) { > PetscCall(DMCreateInterpolation(da_list[k - 1], da_list[k], &R, NULL)); > PetscCall(PCMGSetInterpolation(pc, k, R)); > PetscCall(MatDestroy(&R)); > } > > The more standard alternative to this is to call KSPSetDM() and have the PCMG setup use the DM > to construct the interpolations (I don't know why ex42.c does this construction itself instead of having the KSPSetDM() process handle it but that doesn't matter). The end result is the same in both cases. > > Since PCREDISTRIBUTE builds its own new matrix (by using only certain rows and columns of the original matrix) the original interpolation > cannot be used for two reasons > > 1) (since it is for the full system) It is for the wrong problem. > > 2) In addition, if you ran with ex42.c the inner KSP does not have access to the interpolation that was constructed so you could not get PCMG to to work as indicated below. > > I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1. > > So the short answer is that there is no "canned" way to use the PCMG process trivially with PCDISTRIBUTE. > > To do what you want requires two additional steps > > 1) after you construct the full interpolation matrix (by using the DM) you need to remove the rows associated with the dof that have been removed by the "locked" variables (and the columns that are associated with coarse grid points that live on the removed points) so that the interpolation is the correct "size" for the smaller problem > > 2) since PCREDISTRIBUTE actually moves dof of freedom between MPI processes for load balancing after it has removed the locked variables you would need to do the exact same movement for the rows of the interpolation matrix that you have constructed (after you have removed the "locked" rows of the interpolation. > > Lots of bookkeeping to acheive 1 and 2 but conceptually simple. > > As an experiment you can try using PCGAMG on the redistributed matrix -redistribute_pc_type gamg to use algebraic multigrid just to see the time and convergence rates. Since GAMG creates its own interpolation based on the matrix and it will be built on the smaller redistributed matrix there will no issue with the wrong "sized" interpolation. Of course you have the overhead of algebraic multigrid and cannot take advantage of geometric multigrid. The GAMG approach may be satisfactory to your needs. > > If you are game for looking more closely at using redistribute with geometric multigrid and PETSc (which will require digging into PETSc source code and using internal information in the PETSc source code) you can start by looking at how we solve variational problems with SNES using reduced space active set methods. SNESVINEWTONRSLS /src/snes/impls/vi/rs/virs.c This code solves problem 1 see() it builds the entire interpolation and then pulls out the required non-locked part. Reduced space active set methods essentially lock the constrained dof and solve a smaller system without those dof at each iteration. > > But it does not solve problem 2. Moving the rows of the "smaller" interpolation to the correct MPI process based on where PCREDISTRIBUTE moved rows. To do this would requring looking at the PCREDISTRUBUTE code and extracting the information of where each row was moving and performing the process for the interpolation matrix. > src/ksp/pc/impls/redistribute/redistribute.c > > Barry > > > > > > > > > > > On Jun 30, 2023, at 8:21 AM, Carl-Johan Thore via petsc-users > wrote: > > Hi, > > I'm trying to run an iterative solver (FGMRES for example) with PCMG as preconditioner. The setup of PCMG > is done roughly as in ex42 of the PETSc-tutorials (https://petsc.org/main/src/ksp/ksp/tutorials/ex42.c.html). > Since I have many locked degrees-of-freedom I would like to use PCREDISTRIBUTE. However, this > results in (30039 is the number of DOFs after redistribute and 55539 the number before): > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Matrix dimensions of A and P are incompatible for MatProductType PtAP: A 30039x30039, P 55539x7803 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.19.0-238-g512d1ae6db4 GIT Date: 2023-04-24 16:37:00 +0200 > [0]PETSC ERROR: topopt on a arch-linux-c-opt Fri Jun 30 13:28:41 2023 > [0]PETSC ERROR: Configure options COPTFLAGS="-O3 -march=native" CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" CUDAOPTFLAGS=-O3 --with-cuda --with-cusp --with-debugging=0 --download-scalapack --download-hdf5 --download-zlib --download-mumps --download-parmetis --download-metis --download-ptscotch --download-hypre --download-spai > [0]PETSC ERROR: #1 MatProductSetFromOptions_Private() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:420 > [0]PETSC ERROR: #2 MatProductSetFromOptions() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:541 > [0]PETSC ERROR: #3 MatPtAP() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:9868 > [0]PETSC ERROR: #4 MatGalerkin() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:10899 > [0]PETSC ERROR: #5 PCSetUp_MG() at /mnt/c/mathware/petsc/src/ksp/pc/impls/mg/mg.c:1029 > [0]PETSC ERROR: #6 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #7 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #8 PCSetUp_Redistribute() at /mnt/c/mathware/petsc/src/ksp/pc/impls/redistribute/redistribute.c:327 > [0]PETSC ERROR: #9 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #10 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #11 KSPSolve_Private() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:824 > [0]PETSC ERROR: #12 KSPSolve() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:1070 > > It?s clear what happens I think, and it kind of make since not all levels are redistributed as they should (?). > Is it possible to use PCMG with PCREDISTRIBUTE in an easy way? > > Kind regards, > Carl-Johan > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Mon Jul 31 15:25:30 2023 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Mon, 31 Jul 2023 13:25:30 -0700 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: <87cz0b8ocx.fsf@jedbrown.org> References: <76B6A62E-7D56-4C7F-9302-BDEF6062931B@gmail.com> <87cz0b8ocx.fsf@jedbrown.org> Message-ID: Maybe that example was a jumping point for some of those studies, but it looks to me like that example has been around since ~2012 and initially only touched on SIMPLE, as opposed to addition of SIMPLE into an augmented lagrange scheme. But it does seem that at some point Carola Kruse added Golub-Kahan bidiagonalization tests to ex70. I don't know very much about that although it seems to be related to AL methods ... but requires that the matrix be symmetric? On Fri, Jul 28, 2023 at 7:04?PM Jed Brown wrote: > See src/snes/tutorials/ex70.c for the code that I think was used for that > paper. > > Alexander Lindsay writes: > > > Sorry for the spam. Looks like these authors have published multiple > papers on the subject > > > > cover.jpg > > Combining the Augmented Lagrangian Preconditioner with the Simple Schur > Complement Approximation | SIAM Journal on > > Scientific Computingdoi.org > > cover.jpg > > > > On Jul 28, 2023, at 12:59 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > ?Do you know of anyone who has applied the augmented Lagrange > methodology to a finite volume discretization? > > > > On Jul 6, 2023, at 6:25 PM, Matthew Knepley wrote: > > > > ?On Thu, Jul 6, 2023 at 8:30?PM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > This is an interesting article that compares a multi-level ILU > algorithm to approximate commutator and augmented > > Lagrange methods: https://doi.org/10.1002/fld.5039 > > > > That is for incompressible NS. The results are not better than > https://arxiv.org/abs/1810.03315, and that PC is considerably > > simpler and already implemented in PETSc. There is an update in to this > > > > > > > https://epubs.siam.org/doi/abs/10.1137/21M1430698?casa_token=Fp_XhuZStZ0AAAAA:YDhnkW9XvAom_b8KocWz-hBEI7FAt46aw3ICa0FvCrOVCtYr9bwvtqJ4aBOTkDSvANKh6YTQEw > > > > > > which removes the need for complicated elements. > > > > You might need stuff like ILU for compressible flow, but I think > incompressible is solved. > > > > Thanks, > > > > Matt > > > > On Wed, Jun 28, 2023 at 11:37?AM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > I do believe that based off the results in > https://doi.org/10.1137/040608817 we should be able to make LSC, with > > proper scaling, compare very favorably with PCD > > > > On Tue, Jun 27, 2023 at 10:41?AM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > I've opened https://gitlab.com/petsc/petsc/-/merge_requests/6642 which > adds a couple more scaling > > applications of the inverse of the diagonal of A > > > > On Mon, Jun 26, 2023 at 6:06?PM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > I guess that similar to the discussions about selfp, the approximation > of the velocity mass matrix by the > > diagonal of the velocity sub-matrix will improve when running a > transient as opposed to a steady > > calculation, especially if the time derivative is lumped.... Just > thinking while typing > > > > On Mon, Jun 26, 2023 at 6:03?PM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > Returning to Sebastian's question about the correctness of the current > LSC implementation: in the > > taxonomy paper that Jed linked to (which talks about SIMPLE, PCD, and > LSC), equation 21 shows four > > applications of the inverse of the velocity mass matrix. In the PETSc > implementation there are at > > most two applications of the reciprocal of the diagonal of A (an > approximation to the velocity mass > > matrix without more plumbing, as already pointed out). It seems like > for code implementations in > > which there are possible scaling differences between the velocity and > pressure equations, that this > > difference in the number of inverse applications could be significant? > I know Jed said that these > > scalings wouldn't really matter if you have a uniform grid, but I'm not > 100% convinced yet. > > > > I might try fiddling around with adding two more reciprocal > applications. > > > > On Fri, Jun 23, 2023 at 1:09?PM Pierre Jolivet > wrote: > > > > On 23 Jun 2023, at 10:06 PM, Pierre Jolivet > wrote: > > > > On 23 Jun 2023, at 9:39 PM, Alexander Lindsay > wrote: > > > > Ah, I see that if I use Pierre's new 'full' option for > -mat_schur_complement_ainv_type > > > > That was not initially done by me > > > > Oops, sorry for the noise, looks like it was done by me indeed in > > 9399e4fd88c6621aad8fe9558ce84df37bd6fada? > > > > Thanks, > > Pierre > > > > (though I recently tweaked MatSchurComplementComputeExplicitOperator() > a bit to use > > KSPMatSolve(), so that if you have a small Schur complement ? which is > not really the case > > for NS ? this could be a viable option, it was previously painfully > slow). > > > > Thanks, > > Pierre > > > > that I get a single iteration for the Schur complement solve with LU. > That's a nice testing > > option > > > > On Fri, Jun 23, 2023 at 12:02?PM Alexander Lindsay < > alexlindsay239 at gmail.com> > > wrote: > > > > I guess it is because the inverse of the diagonal form of A00 becomes a > poor > > representation of the inverse of A00? I guess naively I would have > thought that the > > blockdiag form of A00 is A00 > > > > On Fri, Jun 23, 2023 at 10:18?AM Alexander Lindsay < > alexlindsay239 at gmail.com> > > wrote: > > > > Hi Jed, I will come back with answers to all of your questions at some > point. I > > mostly just deal with MOOSE users who come to me and tell me their > solve is > > converging slowly, asking me how to fix it. So I generally assume they > have > > built an appropriate mesh and problem size for the problem they want to > solve > > and added appropriate turbulence modeling (although my general > assumption > > is often violated). > > > > > And to confirm, are you doing a nonlinearly implicit > velocity-pressure solve? > > > > Yes, this is our default. > > > > A general question: it seems that it is well known that the quality of > selfp > > degrades with increasing advection. Why is that? > > > > On Wed, Jun 7, 2023 at 8:01?PM Jed Brown wrote: > > > > Alexander Lindsay writes: > > > > > This has been a great discussion to follow. Regarding > > > > > >> when time stepping, you have enough mass matrix that cheaper > > preconditioners are good enough > > > > > > I'm curious what some algebraic recommendations might be for high Re > > in > > > transients. > > > > What mesh aspect ratio and streamline CFL number? Assuming your model > > is turbulent, can you say anything about momentum thickness Reynolds > > number Re_?? What is your wall normal spacing in plus units? (Wall > > resolved or wall modeled?) > > > > And to confirm, are you doing a nonlinearly implicit velocity-pressure > > solve? > > > > > I've found one-level DD to be ineffective when applied monolithically > or > > to the momentum block of a split, as it scales with the mesh size. > > > > I wouldn't put too much weight on "scaling with mesh size" per se. You > > want an efficient solver for the coarsest mesh that delivers sufficient > > accuracy in your flow regime. Constants matter. > > > > Refining the mesh while holding time steps constant changes the > advective > > CFL number as well as cell Peclet/cell Reynolds numbers. A meaningful > > scaling study is to increase Reynolds number (e.g., by growing the > domain) > > while keeping mesh size matched in terms of plus units in the viscous > > sublayer and Kolmogorov length in the outer boundary layer. That turns > > out to not be a very automatic study to do, but it's what matters and > you > > can spend a lot of time chasing ghosts with naive scaling studies. > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any > > results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 31 17:09:56 2023 From: jed at jedbrown.org (Jed Brown) Date: Mon, 31 Jul 2023 16:09:56 -0600 Subject: [petsc-users] Scalable Solver for Incompressible Flow In-Reply-To: References: <76B6A62E-7D56-4C7F-9302-BDEF6062931B@gmail.com> <87cz0b8ocx.fsf@jedbrown.org> Message-ID: <87wmyf7mx7.fsf@jedbrown.org> I would email the author. He's been helpful in the past and that newer paper may have been extensions that didn't make it into the upstream example. Alexander Lindsay writes: > Maybe that example was a jumping point for some of those studies, but it > looks to me like that example has been around since ~2012 and initially > only touched on SIMPLE, as opposed to addition of SIMPLE into an > augmented lagrange scheme. > > But it does seem that at some point Carola Kruse added Golub-Kahan > bidiagonalization tests to ex70. I don't know very much about that although > it seems to be related to AL methods ... but requires that the matrix be > symmetric? > > On Fri, Jul 28, 2023 at 7:04?PM Jed Brown wrote: > >> See src/snes/tutorials/ex70.c for the code that I think was used for that >> paper. >> >> Alexander Lindsay writes: >> >> > Sorry for the spam. Looks like these authors have published multiple >> papers on the subject >> > >> > cover.jpg >> > Combining the Augmented Lagrangian Preconditioner with the Simple Schur >> Complement Approximation | SIAM Journal on >> > Scientific Computingdoi.org >> > cover.jpg >> > >> > On Jul 28, 2023, at 12:59 PM, Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > ?Do you know of anyone who has applied the augmented Lagrange >> methodology to a finite volume discretization? >> > >> > On Jul 6, 2023, at 6:25 PM, Matthew Knepley wrote: >> > >> > ?On Thu, Jul 6, 2023 at 8:30?PM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > This is an interesting article that compares a multi-level ILU >> algorithm to approximate commutator and augmented >> > Lagrange methods: https://doi.org/10.1002/fld.5039 >> > >> > That is for incompressible NS. The results are not better than >> https://arxiv.org/abs/1810.03315, and that PC is considerably >> > simpler and already implemented in PETSc. There is an update in to this >> > >> > >> > >> https://epubs.siam.org/doi/abs/10.1137/21M1430698?casa_token=Fp_XhuZStZ0AAAAA:YDhnkW9XvAom_b8KocWz-hBEI7FAt46aw3ICa0FvCrOVCtYr9bwvtqJ4aBOTkDSvANKh6YTQEw >> > >> > >> > which removes the need for complicated elements. >> > >> > You might need stuff like ILU for compressible flow, but I think >> incompressible is solved. >> > >> > Thanks, >> > >> > Matt >> > >> > On Wed, Jun 28, 2023 at 11:37?AM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > I do believe that based off the results in >> https://doi.org/10.1137/040608817 we should be able to make LSC, with >> > proper scaling, compare very favorably with PCD >> > >> > On Tue, Jun 27, 2023 at 10:41?AM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > I've opened https://gitlab.com/petsc/petsc/-/merge_requests/6642 which >> adds a couple more scaling >> > applications of the inverse of the diagonal of A >> > >> > On Mon, Jun 26, 2023 at 6:06?PM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > I guess that similar to the discussions about selfp, the approximation >> of the velocity mass matrix by the >> > diagonal of the velocity sub-matrix will improve when running a >> transient as opposed to a steady >> > calculation, especially if the time derivative is lumped.... Just >> thinking while typing >> > >> > On Mon, Jun 26, 2023 at 6:03?PM Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > Returning to Sebastian's question about the correctness of the current >> LSC implementation: in the >> > taxonomy paper that Jed linked to (which talks about SIMPLE, PCD, and >> LSC), equation 21 shows four >> > applications of the inverse of the velocity mass matrix. In the PETSc >> implementation there are at >> > most two applications of the reciprocal of the diagonal of A (an >> approximation to the velocity mass >> > matrix without more plumbing, as already pointed out). It seems like >> for code implementations in >> > which there are possible scaling differences between the velocity and >> pressure equations, that this >> > difference in the number of inverse applications could be significant? >> I know Jed said that these >> > scalings wouldn't really matter if you have a uniform grid, but I'm not >> 100% convinced yet. >> > >> > I might try fiddling around with adding two more reciprocal >> applications. >> > >> > On Fri, Jun 23, 2023 at 1:09?PM Pierre Jolivet >> wrote: >> > >> > On 23 Jun 2023, at 10:06 PM, Pierre Jolivet >> wrote: >> > >> > On 23 Jun 2023, at 9:39 PM, Alexander Lindsay >> wrote: >> > >> > Ah, I see that if I use Pierre's new 'full' option for >> -mat_schur_complement_ainv_type >> > >> > That was not initially done by me >> > >> > Oops, sorry for the noise, looks like it was done by me indeed in >> > 9399e4fd88c6621aad8fe9558ce84df37bd6fada? >> > >> > Thanks, >> > Pierre >> > >> > (though I recently tweaked MatSchurComplementComputeExplicitOperator() >> a bit to use >> > KSPMatSolve(), so that if you have a small Schur complement ? which is >> not really the case >> > for NS ? this could be a viable option, it was previously painfully >> slow). >> > >> > Thanks, >> > Pierre >> > >> > that I get a single iteration for the Schur complement solve with LU. >> That's a nice testing >> > option >> > >> > On Fri, Jun 23, 2023 at 12:02?PM Alexander Lindsay < >> alexlindsay239 at gmail.com> >> > wrote: >> > >> > I guess it is because the inverse of the diagonal form of A00 becomes a >> poor >> > representation of the inverse of A00? I guess naively I would have >> thought that the >> > blockdiag form of A00 is A00 >> > >> > On Fri, Jun 23, 2023 at 10:18?AM Alexander Lindsay < >> alexlindsay239 at gmail.com> >> > wrote: >> > >> > Hi Jed, I will come back with answers to all of your questions at some >> point. I >> > mostly just deal with MOOSE users who come to me and tell me their >> solve is >> > converging slowly, asking me how to fix it. So I generally assume they >> have >> > built an appropriate mesh and problem size for the problem they want to >> solve >> > and added appropriate turbulence modeling (although my general >> assumption >> > is often violated). >> > >> > > And to confirm, are you doing a nonlinearly implicit >> velocity-pressure solve? >> > >> > Yes, this is our default. >> > >> > A general question: it seems that it is well known that the quality of >> selfp >> > degrades with increasing advection. Why is that? >> > >> > On Wed, Jun 7, 2023 at 8:01?PM Jed Brown wrote: >> > >> > Alexander Lindsay writes: >> > >> > > This has been a great discussion to follow. Regarding >> > > >> > >> when time stepping, you have enough mass matrix that cheaper >> > preconditioners are good enough >> > > >> > > I'm curious what some algebraic recommendations might be for high Re >> > in >> > > transients. >> > >> > What mesh aspect ratio and streamline CFL number? Assuming your model >> > is turbulent, can you say anything about momentum thickness Reynolds >> > number Re_?? What is your wall normal spacing in plus units? (Wall >> > resolved or wall modeled?) >> > >> > And to confirm, are you doing a nonlinearly implicit velocity-pressure >> > solve? >> > >> > > I've found one-level DD to be ineffective when applied monolithically >> or >> > to the momentum block of a split, as it scales with the mesh size. >> > >> > I wouldn't put too much weight on "scaling with mesh size" per se. You >> > want an efficient solver for the coarsest mesh that delivers sufficient >> > accuracy in your flow regime. Constants matter. >> > >> > Refining the mesh while holding time steps constant changes the >> advective >> > CFL number as well as cell Peclet/cell Reynolds numbers. A meaningful >> > scaling study is to increase Reynolds number (e.g., by growing the >> domain) >> > while keeping mesh size matched in terms of plus units in the viscous >> > sublayer and Kolmogorov length in the outer boundary layer. That turns >> > out to not be a very automatic study to do, but it's what matters and >> you >> > can spend a lot of time chasing ghosts with naive scaling studies. >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any >> > results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >>